Using OGR to Visualize Data
The biggest excuse I hear time and time again for not using open source geospatial tools is that they are too confusing to use. Take GDAL/OGR for example, the webpage probably scares off most users because of the lack of examples of how to use both toolsets. But the tools themselves are so easy to use this shouldn’t scare anyone off. In fact, if you’ve used ArcInfo to convert a coverage to a shapefile in your past, you have all the skills needed to use both GDAL and OGR.
That said Tom Kralidis has a great example of using OGR to convert a Microsoft Excel file to KML. GDAL/OGR is integral to so many GIS applications (even ESRI uses it) that any GIS professional should have at least a basic understanding to how it works. Look at Tom’s example, download FWTools and get cracking on using GDAL/OGR.
Fans of the Waffle House wait for open source geospatial tool users to create GeoRSS feed of store locations
It’s an odd and convoluted thing – some organizations deliberately dodge Open Source geospatial products in favor of commercial ones, citing support and sustainability concerns, yet on the other hand, you have folks who do strictly OS work, with arrogance and cynicism toward COTS products (sort of like the folks who dogpile ESRI, Microsoft, et al. in critiquing them for every fault, yet in actuality, don’t even use the software themselves).
As such, typically the choice of geospatial tools is generally dictated by what organizational constraints your shop has (e.g. some organizations have their “IT Gestapo”)
Add to this, that most folks are wrapped up in day-to-day production issues, and as such, are generally afforded precious little time and bandwidth to go and research alternatives, particularly if those alternatives might not have a whole lot of documentation (or not easily accessible documentation) or examples (per your OGR example, or Oracle Spatial and others) or if they aren’t immediately ready to use “out of the box” –
Sad, as a lot of opportunity gets missed. As far as the cynicism and arrogance (regardless of which camp it comes from) goes, it must be some unwritten prerequisite in the community.
James, thanks for this website. I found this a few months ago and really enjoy it. As a web developer, once involved with MapInfo/MapExtreme, I have moved away from GIS over the past few years. Looking at the technologies available now, wow! It is time to get back in on the OpenSource angle. Thanks for all your writings!
I for one, really appreciate the hard work done by the OGR/GDAL developers. These guys have done a terrific job putting together one of the most useful libraries I use in my development and consulting work.
I agree docs are scary and everything gets even more complicated when you think about non-English speakers (well tech people are often ok with reading, but 1 out of 100 will dare to ask something in English). I believe spending a minuscule amount of time on writing to illustrate how work with this stuff pays off in numbers of users and efficiency in use. I wrote some tutorials on how to use GDAL/OGR/Rgdal and it seems like people are pretty happy about that.
I completely agree that the GDAL/OGR libraries are really an excellent piece of work. Earlier this year I was involved in a project in which we needed to load data from shapefiles or Personal Geodatabases (PGDB) into PostGIS. The requirements & budget were such that FME was not practical and the lead developer wasn’t sure how to proceed. I introduced him to GDAL/ORG & FWTools. Then we were able to easily cook up an ArcCatalog custom tool that just plain works.
Last year I was struggling with getting decent response times in Manifold or Mapserver with some large county mosaics of NAIP imagery in MrSID format. I tried some conversions with the LizardTech tools but they were slow & ended up stripping the projection information. I wrote a batch command file for FWTools to execute and in a very reasonable amount of time I had convertred the large unwieldy MrSID files into quarter quad J2K’s. With another simple command, I had the Mapserver equivalent of an ESRI Image Catalog. Then suddenly I had a very fast image server were there was once a slow & unresponive service.
My apologies on my tangential and cynical rant above –
To give another perspective, we have been using FME for several years now – and I owe special gratitude to Safe.com for quickly helping us through some issues during Hurricane Katrina – Juan Chu Chow in particular, so I admit I have only scratched the surface of GDAL/OGR – but what I have seen is impressive and robust.
Certainly given the huge (and growing) market demand for spatial ETL tools, GDAL and OGR will certainly be destined to go far.
But documentation and examples certainly help – as do ready-to-use builds for a variety of platforms, and other things such as user interfaces (OGR/GDAL is ready-to-use for a variety of platforms, on a command-line basis).
In crisis circumstances like a Hurricane Katrina, where time is of the essence, GDAL/OGR, and some of the other Open Source tools (PostGIS) are not as accessible and immediately usable as they perhaps could be, particularly when one might end up with a mix of personnel rapidly assembled and assigned on detail for emergency response – they need to ramp up quickly and might not have the amount of support needed. (this is not specific to just GDAL/OGR, but to other parts of the FWTools suite and other OS tools in general)
Certainly use cases similar to ours exist, and probably many folks have already tackled and surmounted some of these questions and they may have already been answered, and while some of the OS gurus have seen, touched or heard the answers – the answers have not always made it back into the broader community via the Wikis, documentation, and so on. I tend to suspect there are far more silent users and developers out there than the visible, core group.
We presently have a number of things coming up in the near future that involve spatial ETL – and don’t yet know if a investment in FME Server will materialize (it’s beyond my own control on this one) – so if we go the OGR route, I would certainly make it my own pledge to contribute back.
Matt – I couldn’t agree more. After struggling with Manifold and other applications to process a larger number of raster datasets, I finally turned to GDAL and some custom programming.
Since v2007, FME also offers very powerful raster processing functions.
You should try them out. They use GDAL for some aspects but you can do all your configuration with the powerful Workbench-application and you do not need any scripting skills.
This is certainly the best way to use these powerful libraries in a userfriendly and well documented way.
Dave, GDAL and OGR need some tutorials for sure, but your claim that they are not very useful in a crisis is false. In the immediate aftermath of Katrina, Joel Schlagel of the USACE wrote to Frank Warmerdam to tell how important the software was to their response. Frank forwarded it to the MapServer list and it’s there in the archive. I’d dig it up for you, but the crappy listserv interface is too annoying.
Sean, I didn’t say (or intend to say) OGR and GDAL couldn’t be useful in a crisis, absolutely they could be.
If a disaster-impacted or disaster-response organization already has folks using OGR and GDAL in their workflows, it would of course be a natural for them.
However, in other instances I have seen in the emergency response realm, people might be pulled from disparate environments, and thrown together on detail with a sudden new and unanticipated need for conversion and transform of data, and not have the requisite familiarity with OGR and GDAL – the associated learning curve might be an issue when time is of the essence.
But this is not at all insurmountable, with good tutorials and documentation – that was the point I was trying to make. I would anticipate that some Emergency Response and other organizations have SOPs for GIS data transform (as we have likewise developed for some of the organizations we work with) – some of which may in fact specifically document workflows and processes using OGR and GDAL – and hopefully some of these get contributed back to the community and get posted at some point.