Massive Datasets - Best Practices?

Roger Edrinn

I'm nearing the end of my project and all the imported shapefiles now have meaning as streams, roads, lakes, forests, parks, etc. plus contours. The roads data consumes 1.3GB of memory, I'd guess the water data is even larger, I've yet to assemble it into a single file.

I'm looking for suggestion for both managing the layers for output and archiving. I can of course create catalog files, like I now have for the contours, however contours are predictable grids. Roads are by county and water is in sub-basin extents. But, catalogs don't help for archiving.

Should I export to gridded shapefiles, does Global Mapper have yet another trick to simplify my life? I need not have the same solution for output and archiving, it would just make life easier. Hard drives and DVD's are so cheap, cost of these is immaterial.

I'm open to suggestions.

Thanks

global_mapper

I would suggest perhaps creating a map catalog covering all of the Tiger Shapefiles of interest (i.e. edges, areas, points). Then do an export to Global Mapper Package files using the Gridding option to generate package files of some size. These should maintain all of the style and attribute information from the features and will likely be smaller than the Shapefiles as well.

Let me know if I can be of further assistance.

Thanks,

Mike
Global Mapper Support
support@globalmapper.com

Roger Edrinn

Forgot one thing. For my roads, I've used edges only and used the vector filters to turn-off the streams and numerous other non-road features. However, for my High-rez water data those filters have to be turned on.

Is their a simple way to strip out all the unwanted lines in the Tiger 2007FE data? The only solution that comes to mind is export to shapefile(s). If shapefiles, I'd probably just grid it to 1 deg squares and catalog those?

Thanks

global_mapper

I would just use the filters to turn off the types that you don't want and export to package files rather than Shapefiles as package files are sure to maintain everything whereas Shapefiles don't have the internal structure to efficiently maintain all settings for a feature.

Thanks,

Mike
Global Mapper Support
support@globalmapper.com

Roger Edrinn

Package files didn't work at all.

I loaded all the water data, areas and lines, as sub-basins. Then I did an export into ~1 deg blocks, 7 columns x 4 rows. I then loaded those 28 package file blocks.

That was my first surprise, during loading it opened all the original shapefiles, several hundred. I was expecting that the package files would be self-contained hydro block with all water data and no connection to the original shapefiles.

I then created a catalog file using all loaded data. Second surprise! I got random red blocks such that when I opened the catalog and zoomed in, their were numerous void areas with no data.

What am I missing/doing wrong?

Thanks

global_mapper

Roger,

The package files are completely self-contained, but they do maintain everything in the original layers from the export. Since your original layer names likely included the SHP file names, it may look like the package file is referring to the SHP files, but they are not, the .shp is just in the layer names.

I'm not sure why you would get gaps. If you zoom into the void areas with the original catalog that the export was done from is there a gap there?

Thanks,

Mike
Global Mapper Support
support@globalmapper.com

Roger Edrinn

global_mapper wrote: »

If you zoom into the void areas with the original catalog that the export was done from is there a gap there?

I didn't export from a catalog, rather a workspace.

Another thing, when I created the catalog from the 28 - 1 deg block, the catalog did NOT contain the names of the 28 blocks, rather a few, but not all, of the shapefiles I was seeing being loaded.

While writing this reply, did another test, same dismal results. Instead of using all loaded, I used "Add Files" and browsed to the road package files. Again I got a few random red blocks and no names of the package files, rather a few shapefies.

However, if I open a single block, it appears to contain ALL the road or water data expected. My problem would seem to be catalog related. When I did a contour catalog, I got a 4 x 7 array of adjoining red blocks.

Now what?

global_mapper

Can you send me your original road workspace that you exported from? I have all of the Tiger 2007 files downloaded so I should be able to try and reproduce your results. Can you also let me know what export bounds that you used for your gridded export so that I can try and exactly replicate your results?

Thanks,

Mike
Global Mapper Support
support@globalmapper.com

Roger Edrinn

The file is 27MB, too big to attach or email. I'm now ftping the file to my website. I'll email you a link when it's finished uploading.

Since the roads do not extend past the border, I'm pretty sure I used the default bounds, 4 rows x 7 columns.

global_mapper

I got the file and exports the 7x4 grid of package files, then loaded those in and created a map catalog and got the same strange boxes that didn't fill in. I did some digging and found a problem adding package files with multiple layers to a map catalog. Once I fixed that everything worked great! Note that the generated package files are fine, it was the map catalog creation from them that was problematic.

I have placed a new build at http://www.globalmapper.com/global_mapper9.zip with the change for you to try. Simply download that file and extract the contents into your existing v9.xx installation folder to give it a try.

Let me know if I can be of further assistance.

Thanks,

Mike
Global Mapper Support
support@globalmapper.com

Roger Edrinn

Good detective work. I too was able to build a voidless catalog with the new build.

Thanks