how to resume mbtiles output

wmiswmwmiswm Global Mapper UserPosts: 220Trusted User
I have a detail map which I need to output to mbtiles level 20. According to the speed of the output, it will take about 80-100 days to complete this single layer. I wonder is there any means that I can pause the output process, shut down GM, reboot the machine, then restart GM, and then resume the output process? The reason I ask is that during this extended period, there will be unavoidable events that I have to deal with, say windows update, whole machine backup etc. If I cannot resume the mbtiles output process in the middle of the run, I very much doubt whether GM can ever output this map for me. Any suggestion?

Best Answer

  • bmg_bobbmg_bob Global Mapper Programmer Posts: 2,032
    Answer ✓
    In the MBTiles Export Options, on the Tiling Options tab, be sure to put a check next to "Skip Existing Tiles", and on the MBTiles Options tab, remove the check next to "Delete Intermediate Tile Files" and always specify the same directory in the Location field. This will allow you to cancel the export, then resume it at a later time.

    It sounds like you are creating detailed tiles for a very large data set. Can you split up your data into smaller exports?

Answers

  • wmiswmwmiswm Global Mapper User Posts: 220Trusted User
    not actually a very big area, but a 1:1000 map of a metropolitan. If I split the whole map into several pieces, the final output will be very inconvenient to use, and there is no way to combine several mbtiles into one single map.

    I can try what you describe and will report back.
  • MykleMykle Global Mapper User Posts: 446Trusted User
    On the same dialog is an option to export only tiles that are complete. The results can be combined with other tiles.
    Mykle
  • wmiswmwmiswm Global Mapper User Posts: 220Trusted User
    great, Bob, GM is working as described. Thank you very much.

    Hi Mykle, would you mind to describe in more detail?
  • wmiswmwmiswm Global Mapper User Posts: 220Trusted User
    hi all, I found a support reply from Mike for an old post: http://forum.globalmapperforum.com/discussion/11459/exporting-large-map-sets

    Quote: You can even run multiple exports at once with the option to skip existing tiles to have multiple cores going at once on the export.

    How can I enable multiple exports at once so as to speed up the export? At the moment only one core of my CPU is working, sadly.
  • MykleMykle Global Mapper User Posts: 446Trusted User
    Hmmm. Had several paragraphs of explanation done, then caught myself. Some of my projects use OSMTiles, and some MBTiles. The OSMTiles result in a directory structure containing individual files for each tile. MBTiles generate one compressed file. I was thinking about OSMTiles, where I can update a small section without generating the entire map. Not so with MBTiles.

    For MBTiles I have a master map covering a large area, but without a lot of detail. Separate MBTile files cover small sections in more detail, but they do not provide continuous coverage (a few are adjacent, but I need to switch files when crossing from one to the other).

    So I was going astray from your question.
    Mykle
  • wmiswmwmiswm Global Mapper User Posts: 220Trusted User
    hi Mykle, I see, thank you very much anyway. As far as I understand, there is no way to combine several mbtiles of different boundaries into one. Different levels of maps of the same boundary can be packed into one however.

    I suppose I cannot cut my area into several smaller pieces to work with. But seeing Mike's old correspondence, I am very interested to his mentioned "multiple exports" option to fully utilize my multicore CPU.
  • bobwillsbobwills Global Mapper User Posts: 48Trusted User
    Hi wmiswm,
    To export multiple times you simply run more than one instance of global mapper - using the same project file and desired export settings.
    I find this works very well with large exported files (tiff) but was unsuccessful using the rmap format as the files are so small each instance seems to overwrite the other as they are not seen in time - but maybe I had a setting incorrectly set.
    I understand there is a modification planned that will use multple cores for this process for a future release.
    regards,

    Brian.
  • wmiswmwmiswm Global Mapper User Posts: 220Trusted User
    I tried the multi thread export and the result is excellent, thank you very much. However, to load two or more instances will require HUGE amount of ram in one computer. And, on top of ram issue, sometimes the individual GM instances will overwrite one another and the efficiency will decrease.

    Instead of outputting the mbtiles by one computer, I would suggest parallel computing. To allow for this, GM please add in the option to specify the range of mbtiles to output. I can then map a network drive and several computers can work in harmony for the tiling process.
  • bmg_bobbmg_bob Global Mapper Programmer Posts: 2,032
    You can accomplish this now. When Global Mapper creates an MBTiles database, it actually generates Tile Mapping Service (TMS) tiles, then imports those tiles into a SQLite data set. Because of this, you should be able to run TMS exports on different computers (with each computer exporting different zoom levels), and specify that the output go to your mapped network drive. Then run the MBTiles export, making sure to 1) skip existing tiles, 2) don't delete temporary tile files, and 3) specify the folder where your TMS files were exported as the temporary file directory. When the MBTiles exporter generates the SQLite database, it will read all of the tiles in the temporary file directory regardless of the number of zoom levels specified, so you only need to specify one zoom level on the MBTiles export in this case.

    What do you mean by "specify the range of tiles"? You can already specify the export bounds and zoom level for which you want to create tiles. Are you looking for something else?
  • wmiswmwmiswm Global Mapper User Posts: 220Trusted User
    Bob, thank you very much for your reply. I am talking about the tiles of ONE level. not one computer for one level and another for another level. My issue is that, in ONE level 20, I have 3 million tiles to generate. This will take me tens of days to finish just this level. I wish to use several computers to work together to speed up the process.

    Why I need several computers? The reason is that for every GM instance, the full set of data will be loaded into the ram. Just for one instance will require 13GB ram. For a "layman" computer, the ram is at maximum 16-32GB. This means, I can load only one or two instances into one computer. If I can use several computers together, I can much speed up the process to just one or two days.

    At the moment, I know that can use as many computers as I prefer, using a net drive as the temporary tile holder. However, the different GM instances may sometimes "overwrite" one another. This will much decrease the efficiency of the output process. If GM allows for specifying the range of tiles to generate (for one level), then the different GM instances will NOT overwrite one another.

    This is a simple solution before the true implementation of multi-thread tile output. Just a slight modification in the current version will do the job.
  • wmiswmwmiswm Global Mapper User Posts: 220Trusted User
    To further clarify, my suggestion has nothing to do with map bound. The map bound of every GM instance should be exactly the same, otherwise the multi-instance workaround will not work.

    What I mean is in say level 20, the jpg range is from 600000 to 3600000. At the moment, if I open 3 GM instances, all the 3 GM instances will work on the same range of jpg tiles 600000 -3600000. The bad thing is that sometimes the GM instances will overwrite one another. Say, the first GM instances just generated the file 638792.jpg, and a split of a second later, the second GM instance, not seeing the result of the first GM instance when it starts its job, will generate the same 638792.jpg tile again and overwrites the file from the first GM instance. The end result is that one cycle of tile generation will be wasted.

    What I suggest is that GM can provide an option to limit the range of output so that the first computer will work on the jpg range 600000 -1600000; the second computer will work on 1600001 -2600000; and the third will work on 2600001 -3600000. When the three computers output to the same folder of the net drive, the tiles will automatically combine into one level.
Sign In or Register to comment.