Mapping Offline
So why bother working offline ? That's life in the field. There are limits to what can be mapped by tracing features off satellite imagery. Ground truthing is required to really know the details of that feature. There is also the activity of mapping an area during an emergency response when there are no cell towers still working. This is very common in wildland firefighting, as we're often offline for the entire 14 day deployment. In some parts of the world, I've been offline for over a month.
It's this second type of activity that requires a fully offline setup. Especially if working in a team with other people that also depend on map data, you may need to produce up to date maps for everyone on a daily basis. Usually on a wildland fire deployment, you are collecting data on helicopter landing zones, staging areas, water sources for fire trucks, etc... Most of that stays in a custom layer as it's temporary data. Some data is good for uploading to OSM though, anything of a more permanent nature is fine. And often you want access to external datasets.
As a climber/backpacker and fire/rescue responder, map data is critical to me. I've seen bad map data wreck an ambulance, get responders lost, or fail to find the resource we need as it's not on the map. I'd rather have no data than bad data.
When I'm field mapping, I can't depend on having access to any data. Fast USB disk drives are cheap these days, so you can carry an amazing amount of data into the field. It may take weeks to download it all, but much of the data doesn't change frequently, like USGS topographical maps. Satellite imagery takes the most time, I just run it as a background job and let it run for days or weeks when I'm in my home office. The server side is usually bandwidth limited anyway.
I have a laptop with lots of memory and fast SSD drives, both of which help when processing data. My personal vehicle has 130 watts of solar panels and a spare battery so I can keep most everything charged. If I'm not in my personal vehicle, power for charging devices gets difficult. I carry a small solar panel which is good enough to keep my phone working, but not the laptop unfortunately. I also carry a big power bank. If I’m lacking reliable charging, I can only collect data, editing has to wait for more power. Some processing of collected data works better on a daily basis while it's fresh in your mind. Most of the time you can edit remotely, and then conflate and upload when you have a cell or wifi connection and some time. I use rural libraries a lot when I find one, most have free wifi. It's much easier than it used to be, most rural towns have some wifi available. Decades ago I'd be using acoustic couplers to a payphone to get online at 300 baud.
To work remotely requires lots of data, so some I keep it on USB drives, and some on my laptop. In my personal vehicle I have a cell booster installed that runs off the spare battery. That'll make a poor connection usable, but doesn't help if there is no cell connection at all. Future plans are to use packet radio over amateur frequencies or the new mobile Starlink to access the map data.
Of course I have my own program for this, but there are multiple other options for downloading tiles. Obviously this part you have to do when online for an extended period. It's a very low cpu load and network bandwidth problem, it's only time consuming. Sometimes downloading raster map tiles can take much more time than you have, so I run a tile server on my laptop. That way I can access it from multiple programs that use TMS. As most of the mobile apps can't handle large files, part of the problem when working offline is chopping huge files into small files for the areas you are working in. It took weeks to download all the map tiles for Colorado, but you only have to do that once.
MobileAtlas is great, and even though I often use my own software, I still use this program a lot. It's easy to use, and open source of course. You can add support for additional online map sources, for example Bing, or ERSI. It doesn't work offline unless you run your own tile server, but I use this for pre-trip mapping if I have the time. Once you have all the tiles downloaded to your desktop or laptop, producing a variety of file formats is easy. I use this to make both basemaps for OsmAnd and mbtiles for ODK Collect. And yes, I often do cache maps so this can work somewhat offline.
There’s a few other options, I stumbled across this simple one the other day, which has a whole bunch of different imagery sources, more than MobileAtlas. (although you can add them)
I use ODK Collect heavily to the point I have my own data collection Xforms oriented to the types of data I collect. Not all is emergency related, more focused on recreational details, like good camping and obscure trails to climbing areas. I use this to test XLSXForm design ideas too, since you need to use those changes in the field to know if you got it right.
The big downside to ODK is that much of the data manipulation is on a server, so it doesn't work offline well. This project contains a Python program that converts the raw XML file ODK Collect saves for data collection, and converts it to the same format ODK Central uses.
ODK does produce an XML data file of the collected data called an instance. You can pull these off your device and then convert them to GeoJson and OSM XML. That way you can then use your normal tools to examine and edit the data. This drops the step of needing ODK Central to do the conversion. I also sync what I'm doing with ODK Central when I can get online, as it's a better tool for keeping track of the workflow.
I edit the XLSForm file using Libreoffice, and use xls2xform to convert it to an ODK XForm. Then I copy to the right place on my device, and it's there. Same for the external data files used for some mapping projects.
To process it offline, I use the odk2csv.py script in this project to convert the XML instance file to the same CSV format used by ODK Central. Then I run the CSVDump.py script to convert it to OSM format. I validate using JOSM, and do any cleanup necessary.
If you have the debugging support for Android enabled, you can copy data files directly over USB. Very useful when offline and updating map data on a mobile device. Once you can copy files to and from the device and work without ODK Central. You might still upload all your submissions when you are back online, but sometimes you want to do some data cleanup while it's fresh in your head.
This part is probably a bit too extreme for most, but I also run a tile server on my laptop for my state of Colorado. This includes USGS topographic maps, but high-resolution sat imagery. This is where having lots of storage comes in handy, but it's much smaller than you would think. A few dozen gig.
I use whatever tile downloading tool I've decided on, and download the region I want. You have to pick smaller regions usually, and repeat the process a lot. Downloading satellite imagery takes weeks, so you do this when online. I often just grab the areas I'm working in, but over time get all the areas in between for total coverage.
Once you have a tile server, you can configure JOSM (or QGIS) to use it. You've already downloaded everything into an ZXY directory structure, so access it as if it was a TMS file. In JOSM go to *imagery preferences*, and then add this example for USGS topographical maps.
tms[10,15]:http://localhost/topotiles/{zoom}/{x}/{y}.png
I also run a postgres database on my laptop as well. Like I mentioned, lots of disk space and a decently performing laptop are critical if you want to really do this right. I have all the OSM data for Colorado loaded, as well as other states and countries I'm doing mapping work in. Much of my software for working on map data uses postgres because it's an efficient way to store a lot of files. If your software uses GDAL, then it's easy to use either a database or a disk file to process. In the field, it's common to be doing daily conflation while details are fresh. It's not unusual to have a restaurant on OSM, but it has a new name and owners.
I'll run ogr2ogr locally to produce small data extracts while traveling, including making mbtiles for ODK Collect just to keep everything fast.