Massive LiDAR data set in map catalog
sant0s
Global Mapper UserTrusted User
in Vector Data
Hey
I downloaded a massive dataset of LiDAR files.
It has more than 30k files, many of them are close to 400MB.
It has more than 30k files, many of them are close to 400MB.
All together are almost 1000 GB...
I know, loading them like that definitly will never work - so the map catalog is the only way.
But even loading a part of that files into the map catalog will take weeks.
Am I am doing something wrong loading the files into the map catalog?
Because its quite straight forwared and I dont know, why it takes forever...
Or is there a limit?
Thanks alot
Answers
-
I am curious to know how you make out on this. One thing I would suggest, if you have not already done, is keep the data local and on your fastest drive. Also, make sure they are all the same projection and datum. Otherwise GM will have to process them first. Also, loading the workspace will take a long time if GM needs to reproject the catalog. So for testing just use geographic. This has been my experience but I think the largest catalog I had was only 10GB and 16000 files. Only 1/100 your size.
-
Jake said:I am curious to know how you make out on this. One thing I would suggest, if you have not already done, is keep the data local and on your fastest drive. Also, make sure they are all the same projection and datum. Otherwise GM will have to process them first. Also, loading the workspace will take a long time if GM needs to reproject the catalog. So for testing just use geographic. This has been my experience but I think the largest catalog I had was only 10GB and 16000 files. Only 1/100 your size.Hey Jake,sry for late answer.What you mean with "...how you make out on this?"Yea, I also think one bottleneck is my HDD.
Its a quick HDD connected over USB3, still it would be better to use the SSD.
But there I dont have enough space.I will try the "geopgraphic" trick, thanks.
The date is all the same - and I click "use same Projection for all" when loading.There is one thing I realised:When adding 1000 files to the map cataloug, it will take forever to load only 100 of these 1000 files.If i only add 100 files, it will run super quick through all these 100 files.So my workflow at the moment is adding always 100 files max at the time and repeat that until I have all my 1000 files.That is kinda annoying, but it works way faster.Adding all 30k files that way does not really workThe LiDAR process itself after everything is loaded works quite good.On the example picture I added 200+ LiDAR files (each between 200MB and 300MB), exported them as 2 massive LiDARs (ground points 3GB + vegetation points 13GB) and processed them in Houdini.In Houdini I had 1.3 BILLION points and I could convert them very quickly into a heightfield.
-
"How you make out on this", meaning if you figured out a solution or workaround. Which it looks like you did.
The 100 file bottleneck might be due your ram being maxxed out. Easy to check that while it is running. If it happens at 100 files despite file size then that is odd and something support should be made aware of.
I script my map catalog creation since we update the source files monthly. If you have not tried take a look at creating GMS files. https://www.bluemarblegeo.com/knowledgebase/global-mapper-21/Scripting_Reference/ScriptReference.html
Those point clouds are amazingly detailed. Very nice. -
I have to load some LiDAR files later, I will check the used RAM - but I do have 128GB of RAM, so most of the stuff runs fine. I will give you feedback about the RAM.Thank you for the Link to the ScriptReferences. Gonna check that out.YesIts amazing how much details these files bring.
On the examples I modified the final Terrain in Houdini with the Heighfield Erode etc.
Its not realistc yet, specially the water masks.But Thueringen has a massive dataset of LiDAR, check it out:Thx for the quick reply!
-
Hey Jack,was just loading 100 LiDAR files.First I copied these files to my quick SSD 960 Pro.RAM usage while loading is between 250 and 800 MB - so all good here.CPU is max 20% (3,8 GHz).Time to load these 100 files from my SSD was about 25 Minutes.To load all the LiDAR files into map catalouge it would take me more or less 4 daysIs that normal since its lots of data or are there ways, to load data quicker?
Maybe there is a way, to load only these files, that are seen in the viewpoert?Of course, I dont need all 20k LiDAR files at the same time.But if I load the files only for the area I am just working on, I always have to manualy search for the tiles I need and that takes alot of time... -
Good questions. Perhaps try what you are doing with a script. You could also break the catalogs into smaller areas.
Categories
- 12.7K All Categories
- 5.6K Features Discussion
- 342 Downloading Imagery
- 1.3K Elevation Data
- 380 Georeferencing Imagery Discussion
- 628 GM Script Language
- 53 User Scripts
- 113 GPS Features
- 414 Projection Questions
- 819 Raster Data
- 1.3K Vector Data
- 6.6K Support
- 177 Announcement and News
- 908 Bug Report
- 558 SDK
- 1.2K Suggestion Box
- 3.7K Technical Support
- 562 Other Discussion
- 129 GIS Data Sources
- 27 Global Mapper Showcase
- 233 How I use Global Mapper
- 107 Global Mapper Forum Website