Yes, after a user started an app on his device the app will be available offline.
The created apps run natively and not as a web app on users' devices. Therefore no permanent internet connection is necessary. Especially with calculators that need to be served in areas with poor network coverage, this is a good scenario for Open as App.
Make sure you have Open as App installed, you are logged in, and the desired app has been launched on the device at least once before you "go offline".