› ›

Massive LiDAR data set in map catalog

sant0ssant0s Global Mapper UserPosts: 27Trusted User
in Vector Data
Hey :)

I downloaded a massive dataset of LiDAR files.
It has more than 30k files, many of them are close to 400MB.
All together are almost 1000 GB...

I know, loading them like that definitly will never work - so the map catalog is the only way.

But even loading a part of that files into the map catalog will take weeks.

Am I am doing something wrong loading the files into the map catalog?
Because its quite straight forwared and I dont know, why it takes forever...

Or is there a limit?

Thanks alot :)

Best Answer

  • JakeJake Global Mapper User Posts: 276
    edited May 2020 Answer ✓
    I am curious to know how you make out on this. One thing I would suggest, if you have not already done, is keep the data local and on your fastest drive. Also, make sure they are all the same projection and datum. Otherwise GM will have to process them first. Also, loading the workspace will take a long time if GM needs to reproject the catalog. So for testing just use geographic. This has been my experience but I think the largest catalog I had was only 10GB and 16000 files. Only 1/100 your size.

Answers

  • sant0ssant0s Global Mapper User Posts: 27Trusted User
    Jake said:
    I am curious to know how you make out on this. One thing I would suggest, if you have not already done, is keep the data local and on your fastest drive. Also, make sure they are all the same projection and datum. Otherwise GM will have to process them first. Also, loading the workspace will take a long time if GM needs to reproject the catalog. So for testing just use geographic. This has been my experience but I think the largest catalog I had was only 10GB and 16000 files. Only 1/100 your size.

    Hey Jake,
    sry for late answer.

    What you mean with "...how you make out on this?" :)

    Yea, I also think one bottleneck is my HDD.
    Its a quick HDD connected over USB3, still it would be better to use the SSD.
    But there I dont have enough space.

    I will try the "geopgraphic" trick, thanks.
    The date is all the same - and I click "use same Projection for all" when loading.

    There is one thing I realised:
    When adding 1000 files to the map cataloug, it will take forever to load only 100 of these 1000 files.
    If i only add 100 files, it will run super quick through all these 100 files.
    So my workflow at the moment is adding always 100 files max at the time and repeat that until I have all my 1000 files.
    That is kinda annoying, but it works way faster.
    Adding all 30k files that way does not really work :D

    The LiDAR process itself after everything is loaded works quite good.

    On the example picture I added 200+ LiDAR files (each between 200MB and 300MB), exported them as 2 massive LiDARs (ground points 3GB + vegetation points 13GB) and processed them in Houdini.
    In Houdini I had 1.3 BILLION points and I could convert them very quickly into a heightfield.



  • JakeJake Global Mapper User Posts: 276Trusted User
    "How you make out on this", meaning if you figured out a solution or workaround. Which it looks like you did.  :)

    The 100 file bottleneck might be due your ram being maxxed out. Easy to check that while it is running. If it happens at 100 files despite file size then that is odd and something support should be made aware of.

    I script my map catalog creation since we update the source files monthly. If you have not tried take a look at creating GMS files. https://www.bluemarblegeo.com/knowledgebase/global-mapper-21/Scripting_Reference/ScriptReference.html

    Those point clouds are amazingly detailed. Very nice.
  • sant0ssant0s Global Mapper User Posts: 27Trusted User
    I have to load some LiDAR files later, I will check the used RAM - but I do have 128GB of RAM, so most of the stuff runs fine. I will give you feedback about the RAM.

    Thank you for the Link to the ScriptReferences. Gonna check that out.

    Yes :D
    Its amazing how much details these files bring.
    On the examples I modified the final Terrain in Houdini with the Heighfield Erode etc.
    Its not realistc yet, specially the water masks.
    But Thueringen has a massive dataset of LiDAR, check it out:


    Thx for the quick reply! :)

    Jake
  • sant0ssant0s Global Mapper User Posts: 27Trusted User
    Hey Jack,

    was just loading 100 LiDAR files.
    First I copied these files to my quick SSD 960 Pro.
    RAM usage while loading is between 250 and 800 MB - so all good here.
    CPU is max 20% (3,8 GHz).
    Time to load these 100 files from my SSD was about 25 Minutes.

    To load all the LiDAR files into map catalouge it would take me more or less 4 days :D

    Is that normal since its lots of data or are there ways, to load data quicker?
    Maybe there is a way, to load only these files, that are seen in the viewpoert?

    Of course, I dont need all 20k LiDAR files at the same time.
    But if I load the files only for the area I am just working on, I always have to manualy search for the tiles I need and that takes alot of time... :D

  • JakeJake Global Mapper User Posts: 276Trusted User
    Good questions. Perhaps try what you are doing with a script. You could also break the catalogs into smaller areas.

Sign In or Register to comment.