*.bag file issues

RossOverstreetRossOverstreet Global Mapper UserPosts: 30
edited January 2011 in Bug Report
Hi Guys,

Here's a link to a BAG from NOAA NGDC
http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H10001-H12000/H11879/BAG/H11879_50cm_1of6.bag.gz

It's 28MB gzipped and 1.23GB unzipped into a *.bag.

I'm using GM12 Beta 2.

I try *.gz and I get the "Error loading file" warning and nothing opens. Error message shown below.

Error loading file H11879_50cm_1of6.bag.gz
Error loading file.
Read failed
HF2Overlay.cpp - 440
Version: v12.00 Beta 2
Build Time: Sep 14 2010 22:31:23


I try the unzipped *.bag and I get "exception occurred while parsing the metadata in the bag file" warning and then it opens properly.

QUESTION: Should GM be able to open *.gz files?

Try this one:
http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H10001-H12000/H11879/BAG/H11879_2m_2of6.bag.gz

Unzip it. Try to load on top of the file loaded above and I get "Unknown error loading HDF5 file".

Close GM12b2, reopen, try loading the "2of6" unzipped bag file by itself and it loads fine this time.

Try loading the "1 of 6" file above along with it and I get "exception occured while parsing metadata" and then "unknown error while loading HDF5 file".
Looks like I can only open 1 bag at a time.

Try this one:
http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H10001-H12000/H11879/BAG/H11879_4m.bag.gz

Close GM12b2, reopen, it loads the unzipped bag fine.

Try loading the "2of6" file along with it and it works just fine.

Try loading the "1of6" file and I get the "error loading HDS5 file again.

The "1of6", "2of6" thing is new naming convention that NOAA is moving to. GM can apparently only open one of those at a time. I can open the old naming convention files along with the new stuff.


---- go to a different survey ---

Get these two files. Unzip them. GM12b2 will open either of them by itself but will not open the second after the first is loaded.

http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H10001-H12000/H11880/BAG/H11880_150cm_MLLW_2of8.bag.gz
http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H10001-H12000/H11880/BAG/H11880_2m_MLLW_4of8.bag.gz

Go get this one:
http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H10001-H12000/H11880/BAG/H11880_1m_MLLW_3of8.bag.gz

It won't open. Give the "unknown error loading HDF5" message.

Go get this one:
http://surveys.ngdc.noaa.gov/mgg/NOS/coast/H10001-H12000/H11880/BAG/H11880_4m_MLLW_5of8.bag.gz

It'll open if you open it in an empty workspace but it won't load after the other data. You can open the "2m_4of8" after it but you can't load the "4m...5of8" after the 2m.

I don't know if NOAA is breaking the rules on the bag file standard or if GM has a bug but there's a problem somewhere.

Here's an example of how NOAA NGDC is listing the data products from its surveys now. It uses a new naming convention that they apparently just moved to.

http://www.ngdc.noaa.gov/nndc/struts/results?op_0=eq&v_0=H11880&t=101523&s=3&d=5&d=8&d=23&d=10&d=24&d=11&d=9&d=12&d=13&d=14&d=15&d=16&d=17&d=18&d=20&d=21&d=19&no_data=suppress

Thanks for any help.

Ross

Comments

  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited September 2010
    Ross,

    I think what is happening is that the BAG data is taking up too much memory, so you can load it all at once as BAG data is currently read entirely into memory at load time. If you have 64-bit Windows and can run the 64-bit version of GM12b2 then you should be able to load a lot more at once.

    I was able to add support for loading the data directly from .bag.gz files so you don't have to extract them yourself. I also added better detection and reporting of out of memory conditions. I have placed a new build at http://www.globalmapper.com/global_mapper12.zip with the change for you to try. Simply download that file and extract the contents into your existing v12.xx installation folder to give it a try. If you are using the 64-bit v12 version there is a new build at http://www.globalmapper.com/global_mapper12_64bit.zip .

    What I would suggest is using the File->Batch Convert/Reproject menu command to batch convert your .bag.gz files to a better format, like Global Mapper Grid, so that you can load all of them at once without using so much memory.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • RossOverstreetRossOverstreet Global Mapper User Posts: 30
    edited September 2010
    Mark,

    Thank you for the help. It's behaving much better now. I'm still on Vista 32bit at the moment.

    I looked that the Batch Convert options and didn't see bag listed as one of the input file options.

    The size differences of the different files are amazing. Here's one example.
    H11882_1_5m.bag.gz
    H11882_1_5m.bag
    H11992_1_5m.gmg
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited September 2010
    The format would be listed as HDF5 in the batch conversion dialog.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • RossOverstreetRossOverstreet Global Mapper User Posts: 30
    edited September 2010
    Opps, meant to write

    The size differences of the different files are amazing. Here's one example.
    H11882_1_5m.bag.gz 3.7MB
    H11882_1_5m.bag 1.5GB
    H11992_1_5m.gmg >1MB

    The gmg is 1500X smaller than the bag and looks the same in GM. I need to study up on bag but it sure seems like a wasteful format at first glance.

    Thanks,
    Ross
  • Frank2Frank2 Super Moderator Posts: 105
    edited September 2010
    Opps, meant to write

    The size differences of the different files are amazing. Here's one example.
    H11882_1_5m.bag.gz 3.7MB
    H11882_1_5m.bag 1.5GB
    H11992_1_5m.gmg >1MB

    The gmg is 1500X smaller than the bag and looks the same in GM. I need to study up on bag but it sure seems like a wasteful format at first glance.

    Thanks,
    Ross

    and earlier:
    Here's a link to a BAG from NOAA NGDC
    http://surveys.ngdc.noaa.gov/mgg/NOS...cm_1of6.bag.gz

    It's 28MB gzipped and 1.23GB unzipped into a *.bag.

    That's because in the above file for example, there are over 165 million sample points (at 4 bytes each), with something like 95% of them void. There's also another statistics (uncertainty) channel of the same size that I'm not loading. When you don't compress that kind of information, it seems bloated. But that's the true kind of memory impact it has, rather than the extremely small size it packs to.
  • Frank2Frank2 Super Moderator Posts: 105
    edited September 2010
    I'll look into these issues early next week at the latest. Right now I'm in debugging hell with a new file format, and I want to keep my concentration on that for a bit longer.
  • RossOverstreetRossOverstreet Global Mapper User Posts: 30
    edited September 2010
    Hi Frank,

    Thanks for the explanation. Although it would be fun to be able to work with the larger bag files, this is NOT a time critical project for me. No hurry.

    I verified that the batch convert from zipped bag file to gmg works as long as I have enough memory to hold the unzipped bag. I'm still in trouble with the really big files. A nifty feature would be the ability to convert the bag files larger than my memory limit into gmg files.

    I really appreciate the exceptional support that you and Mike provide.

    Thank you,
    Ross
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited September 2010
    Ross,

    We should be able to add support for any size BAG file even in the 32-bit version, we just need to look at it a bit.

    As you noticed, the GMG format is about as good as you can get for compressing terrain data. It uses a bunch of different algorithms on tiles of data to determine how best to compress each tile. Large areas of the same value (like void values) will compress to almost nothing.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • RossOverstreetRossOverstreet Global Mapper User Posts: 30
    edited November 2010
    Hi Guys,

    I'm just wondering if support for converting large *.bag files is still on the "to do" list or if some progress has been made on it in one of the new releases. I'm currently running the Oct 5 2010 build of GM12 on a 32 bit Vista computer. I would like to play around with some bag files are too large for 32-bit memory.

    Happy Thanksgiving to all!

    Thanks,
    Ross
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited November 2010
    While it is still on our todo list for 32-bit versions, the 64-bit version of Global Mapper should already allow very large BAG files to be loaded, so if you have access to a 64-bit Windows machine you could use it for those very large files.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • limeyxlimeyx Global Mapper User Posts: 12
    edited January 2011
    Hi,

    Just wondering if there was ever any resolution to this. I have GlobalMapper 12.01
    64 bit version and am having the same issues opening or Batch Converting BAG files...

    I have a laptop with 4G of RAM.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited January 2011
    There haven't been any recent changes, although I would expect most BAG files to load in the 64-bit Global Mapper release. How large is the BAG file that you are trying to load?

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • limeyxlimeyx Global Mapper User Posts: 12
    edited January 2011
    There haven't been any recent changes, although I would expect most BAG files to load in the 64-bit Global Mapper release. How large is the BAG file that you are trying to load?

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com

    Uh wow, its 14G file. I thought it was 14M given that the .gz version of the download was around that size.
    What a stupid file format.

    EDIT: I am trying with h5dump and various related tools to get something to work but this is a rats-nest that seems to have no end...
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited January 2011
    Yikes that is a massive file! If you have a 64-bit system available with enough memory (or you can check up your virtual memory high enough) then you could convert it to something more reasonable, like a Global Mapper Grid file, using Global Mapper, but I'm afraid the challenge is going to be getting the darn thing loaded first.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • limeyxlimeyx Global Mapper User Posts: 12
    edited January 2011
    Yikes that is a massive file! If you have a 64-bit system available with enough memory (or you can check up your virtual memory high enough) then you could convert it to something more reasonable, like a Global Mapper Grid file, using Global Mapper, but I'm afraid the challenge is going to be getting the darn thing loaded first.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com

    Indeed, and the thing is I think mostly zero/empty entries.

    h5dump and friends are ... well, not "friends" and dont seem very useful.

    i do have the option to get raw data which is much smaller and can be processed with MBSystem but this seemed like a really nice option.

    How a 17M file can reasonably go to 14G I just dont know
Sign In or Register to comment.