The Holy Grail

November 9th, 2007 · 13 Comments · ArcSDE, ESRI, FeatureServer, GDAL/OGR, Open Source

This is clear to me:

FeatureServer + ArcSDE Data Store = Holy Grail

Being able to store data in ArcSDE, but still access it freely across any and all platforms. ArcGIS clients can hit ArcSDE and everyone else can enjoy data served by FeatureServer. But deep down this is even bigger than FeatureServer because really what I’m talking about is GDAL/OGR ArcSDE Vector Write Support. That opens up loads of open source projects to ArcSDE users and gives them the best of all worlds. ArcGIS Desktop/Server users can continue using ArcSDE they way they always have and open source solutions can come right in the front door and coexist with existing workflows. The benefit will be realized by users who will be able to get products that work best for them.

If I have to sell Amway door to door, I will get this project funded.

The gatekeeper will no longer be able to stop users from writing to ArcSDE from OGR.

Tags: amway·arccatalog·arcgis·ArcGIS Desktop·ArcGIS Server·arcmap·ArcSDE·ESRI·FeatureServer·GDAL·GIS·ogr

13 responses so far ↓

1 Erin // Nov 9, 2007 at 4:10 pm

If ESRI uses GDAL/OGR in their projects, you think they would have given back to the community by writing/funding this themselves, no? I mean I realize that ESRI doesn’t like things not invented there, but this seems like something ESRI should have done years ago in the sense that ArcSDE should be accessable by everyone.
2 Brian Timoney // Nov 9, 2007 at 4:28 pm

Bingo.

A lightweight open-source web tier accessing SDE adds a number of attractive cost-effective options for integrating with a wide variety of front-ends.

While we’re on the subject, the latest version of MapServer 4 Windows (MS4W), has ArcSDE bindings built in. So for those frustrated with ArcServer’s WMS behavior with non-Arc clients, there’s a nice alternative there as well.

BT

p.s. And don’t overlook the possibilities with FDO….
3 Anon // Nov 9, 2007 at 4:32 pm

BT, I think James is talking write not read ArcSDE, but you are correct. The ability to push data to ArcSDE sounds like a wonderful idea to me.
4 Bill // Nov 9, 2007 at 7:58 pm

The ArcSDE C and Java (not ArcObjects) libraries offer possibilities here. For instance, GeoServer uses the Java lib to enable ArcSDE as a data store. These libraries also enable writes. The same applies to the C library. It’s not pretty but maybe wrapping them would get you there.
5 KoS // Nov 10, 2007 at 9:11 am

Not sure what time James made the post yesterday.

I think it’s slight ironic coincidence there is a pic from Wiz of Oz. And last night on TBS or TNT, Wiz of Oz was on.

The chicken before the egg or egg before the chicken?

KoS
6 Tim Maddle // Nov 10, 2007 at 10:12 am

James, how is the performance? What, in general terms, are the sizes of the datasets you’re working with? Where I work, my problem is that ArcSDE is a tool that an administrator loves but a user hates. Opening layers from the SDE in ArcMap is slower as compared to opening Shapefiles or PGDB. Both my ArcIMS and ArcGIS Server services run considerably better when the source is a Shapefile or PGDB as compared to the SDE.

Of course, it’s hard for me to know for certain if this is a situation specific to my organization or a known trade-off of using the SDE. I’ve used MapServer and indexed shapefiles to achieve fantastic performance, but my organization frowns on the use of shapefiles, so I’m trying to figure ways to get the data from ArcSDE in a fast fashion. I went to the UC, and probably the most exciting thing I saw there was the ability to run spatial SQL queries against ArcSDE. My hope is that this will be a much faster way to serve data from the SDE rather than going through an ArcIMS/ArcGIS Server process.

On another note, you’ve got me worked up about OpenLayers. I’ve had a bundle of fun with VE and intend to continue using it, but I think OpenLayers may be a better fit for a couple of specific pages in my current main project.
7 Tim Maddle // Nov 11, 2007 at 5:48 am

On the OpenLayers side, this is a link from Dave Bouwman’s blog with some OpenLayers examples:

http://www.openlayers.org/dev/examples/

Take a look if you are interested in OpenLayers.
8 Jody Garnett // Nov 12, 2007 at 2:08 am

Well the Java side is in pretty good shape (if only you guys would redistribute your jars so we can compile). Right now we reverse engineer *just* the API so we can keep our builds going - this leaves us the ability to ship open source solutions (just add drop the jar from ESRI in magic folder X).

That said the hack does work; drop GeoServer or uDig into your workflow today GeoServer especially will open up the gates of the web and allow OpenLayers and all the rest to play.
9 J Wallis // Nov 12, 2007 at 2:39 pm

“and open source solutions can come right in the front door”

Yeah….that is why we keep the front door locked. If we didn’t we’d have every Tom Dick and Harry consultant coming in here to peddle their wares.
10 Jody Garnett // Nov 12, 2007 at 3:12 pm

Depends what hat you want us to wear; my comments above are with an open source hat.

The developer communities I support are given to bouts of ArcSDE development. Right now I have to ask volunteers to go through several extra steps - and for the open source card to work I *need* those volunteers (quality depends on them).

It is a fairly common situation; some vendors have turned things around such as DB2. Others don’t get it like Oracle. We have spoke to ESRI about this tension in the past, and I would like to encourage progress.
11 J Wallis // Nov 12, 2007 at 8:36 pm

Actually, Oracle gets it better than anyone. And now Microsoft will soon “get it” when SQL 2008 comes out. “Place” just needs to be another data type in the database along with nvarchar, long int, and the rest of the family.
12 Matt Priour // Nov 13, 2007 at 4:05 am

There is already an implementation of this idea using 100% ESRI backend to serve & process the data and OpenLayers as the front end, visualization & editing platform.
I’ve seen at least 2 threads by OpenLayers users talking about such projects & implementations.
Using FeatureServer just means that you no longer even need the ArcGIS Server components.
13 Tim Maddle // Nov 14, 2007 at 7:11 pm

…which has huge ramifications. That means that an organization looking to setup an additional spatial server does not need to buy another ArcGIS Server license. Right now, I think ESRI benefits from a dearth of knowledge of the alternative means to store and publish GIS data. There are certainly a number of technies who know about products such as GDAL, Mapserver, GeoServer, etc., but there are plenty of decision-makers who believe its the “ESRIway” or the highway. I know that I’m witnessing that
situation firsthand.

Responding to my post about ArcSDE and performance, I found the time to connect to ArcSDE using MapServer via GDAL. At first I was encouraged, as performance was great with some of our smaller layers, but things really slowed down when trying to create images based on some of our larger point layers, but there’s still tons of testing to do.