The Big Picture of ArcGIS Server
OK so how does all this now ESRI server stuff fit into the big picture? Well lets take Declan Butler’s wonderful avian flu outbreak informationn map. Declan used ArcGIS to create his map and then Arc2Earth to export that map to KML. This works great when for Declan because he’s just using this as an example/demo, but if you really want to share up to the minute information with digital earth clients, this model breaks down because it only gets updated when Declan manually uses Arc2Earth. Now lets say Declan had a nice big server running ArcGIS Server, he could serve this data up in a couple ways.
First off he could use the new cartographic tools in 9.2 to make a very visually pleasing map and serve up the the ArcGIS Server service for ArcGIS (ArcGIS Desktop, ArcGIS Explorer, Web ADF) clients (both desktop and web) to consume (because if anyone has ever seen an ArcGIS Server demo, “rich clients” consume). Anyway that covers your ESRI users, but what about the rest of the world? Well you can offer up a KML network link that Google Earth users (World Wind and others too) can connect to. This KML won’t be as visually pleasing or as full featured as the AGS service, but it will be usable by many more people. In addition to the normal AGS service and the KML service, you can also offer up a WMS service for other users who would like the WMS option. So from one server product, you can cover almost all the spectrum (except for those people who want WFS, you can talk to Jeremy). And because it is “live”, you don’t need to worry about exporting it out every time there is a change.
OK so that picture is a little idyllic. I have no idea how much hardware would be needed to accomplish such a task and of course Declan on his own could never afford the hardware/software requirements for such an endeavor. Still if you think about how many organizations will now have access to ArcGIS Server because of the changes to the licensing, you can only hope that users of this new technology will want to share their data via KML, WMS or AGS for the community to integrate into their digital earth clients.
The days of using the standard ArcIMS HTML client are over and I think even to just assume the community at large will be happy with the new web ADF on its own is shortsighted. Giving people a choice of how to consume your services is the best way to ensure people actually use them.
You could say, James, that you need to get “naked.” “Application naked” with your data and your services. Right?
That is one way to look at it. Another way is to say you want all your services to be both platform and client agnostic. Exposing these services is a great way to start, but also offering services that can be used by anyone anywhere is also important.
Yet another great post James. This is the kind of stuff the ESRI product managers should be posting.
Your blog is such a good read, keep up the great work.
James:
While I by no means speak for Declan, we’ve had conversations about presenting datasets by integrating PostGIS with the Google Earth interface. In addition to easy uploads of new data via simple web forms, the Network Links allow for immediate broadcast of the data or a user could choose to download the data as a standalone KMZ file (which with the new Regions functionality could contain a hefty quantity of features). Going the WMS route, you could throw a MapServer instance that hits on the same PostGIS database. As for geoprocessing, one could use PostGIS’ native capabilities and provide buffering, intersecting, and distance querying tools. (We demonstrate those capabilities at our GulfImpact.com site).
How much does it cost? Well, our good friends at HostGIS will set you up with your own dedicated server with a full suite of open source apps and you pay a $300 setup fee plus $100-$170/mo based on hardware/transfer needs. And Gregor provides top drawer support. That’s a very attractive value proposition for us at least.
If ArcServer hastens the demise of ArcIMS, then I’m all for it. However a world with multiple ways to consume KML and WMS has many compelling back-end possibilities outside the ESRI stack.
BT
Brian,
This post is by no means any comparison of ArcGIS vs open source by any means. Those who do use ESRI now can serve up maps that can be used by many clients. I don’t think in Declan’s case that he’d even need to go the route I suggest and only used his project as an example as just about everyone knows it.
My point Brian was that those working within the ESRI world now have a way to produce KML and WMS that can be consumed outside the ESRI suite. That is a huge shift for many ESRI users and one that must be pushed for the greater good of us all.
One does not need to use ArcGIS Server and I suspect when the hardware/software costs are totaled, many will not be able to go that route. Still those who have invested millions in ESRI software can now become part of the greater community with ArcGIS Server where before they might not have ever had that chance.
I have to agree, great post!
Great post,
I don’t know if I am totally happy with giving up ArcIMS API. I like the new look and feel, but the power to create client rich applications through the ArcIMS API will be extremely lacking. I have not spent much time looking at the ADF yet or programming my custom logic into GIS Server.
From one session on ArcIMS I attended at the EUC it appears the API will still be there and AIMS will be a publishing branch for ArcGIS server. At times, you don’t want to create maps to share, you just want to create GIS enabled portions of a larger application, which ArcIMS is perfect for. I know… I am preaching to the choire… lets see, where is the email address for the ArcIMS development staff…….
Anyone know any good examples of organizations developing/using ArcServer applications?
One thing I’d say about the ADF is that it’s really cool, but unless ESRI has changed their tune in the last 2 weeks, you’ll pay a per-processor licensing fee on the box where the ADF run-time is deployed. So – suppose you want to use your pricy 2 socket AGS license to it’s max – you’ll put the web site(s) on a separate machine. But maybe that’s a two socket system – do I need to buy 2 more sockets of that pricy AGS licensing? This is still unclear to me.
As I said, it’s a nice ADF, but it’s hella complex, and still uses DCOM under the hood to talk to the AGS box (creates issues crossing fairewalls). Thus, if I’m also supposed to pay a per-CPU fee to use it, maybe I look at rolling my own out of OpenLayers and the Prototype js stack, and make calls back to AGS web services or WMS? Sure, it would not have the “manager” but for serious web developers, that’s not an issue. It would however be open and free. I bring up the “open” thing because ADF will ship with bugs – that’s just reality. ESRI’s track record for fixing bugs in their web api’s is poor at best. (Note the multi-year lag in a native .NET ArcIMS object model! Many developers simply wrote their own) Thus, I would implore ESRI to ship the source for the ADF controls so developers can make the changes, rather than waiting for a patch. If this is not the case, expect an onslaught of support calls related to the ADF controls not playing nice in certain situations, and a barrage in forums and blogs.
Re: AGS apps in the wild – there are very few at this point. It’s mainly b/c at 9.1 the price/performance ratio is not there to support an internet audience. Even then, all our clients have used it via web services – mainly to run analyses or create cartographic products which ArcIMS can not. Despite the hype, I really don’t expect this to change a whole lot – with the exception of organizations that have Enterprise Licensing agreements – as they will not be constrained by the per-socket costs.
you have to pay for the ADF now? I thought that was free?
James, dumb question: why can’t Declan simply script Arc2Earth to execute hourly, or whip up a Python/Ruby (Perl, if you must) script to serialize his data as KML? Seems like that’s exactly what he needs.
hey Sean
you can script A2E using the built in Geoprocessing tool (callable from any python script or model) or you could use the simple A2E interface to export from your own code. However, maybe what James is getting at is the larger sets of data. Declan’s avian flu data is rather small when compared to some of others, so maybe its not a good example. In the other cases, a server streaming data directly into GE works much better. It should also be noted that AGS 9.2 and AGX will not support KML 2.1 when released (they are waiting till GE4 is out of beta?!?). IMO, the need for a server is most appropriate when large amounts of data is present and without the Region support in KML 2.1, you are going to run into all of the problems with large datasets that are present today in GE3. so, I guess YMMV.
As I said Sean, there is no reason for Declan to use ArcGIS Server. I only used his example because everyone knows about it, he is using ArcGIS and he is exporting it manually. Just because one can use ArcGIS Server doesn’t mean they should and I wasn’t implying that would be the case.
What I was saying is those who do use ESRI now have an easy way to share their data.
Hi guys
I’m on the typical long French August vacation in the countryside, but I couldn’t resist a brief, unthought-through, contribution to this discussion. James is of course spot on, and I’m actually discussing this exact course of action with senior ESRI staff. Brian Timoney and Brian Flood are also correct in their observations.
Brian Flood pertinently points out that my datasets, while relatively large by GE standards, are small by GIS standards. There are also many additional GIS analytical datasets that one could add to my relatively simple visualization to take it further.
There are a couple of aspects here:
1. The basic flu datasets show how by databasing and structuring available unstructured information from international agencies, one can quickly add some added value (plus GE functionality), and do better mapping than these agencies had done before, just by thinking through the need for stuctured data, and visualization possibilities.
As James says, the real-time ArcGIS server option is the ideal, and I have been frustrated by the batch manual update method I use — hence my discussions with ESRI and Brian Timoney. But getting the data, and producing structured data, remains the most important basic compoment, in particular in scientific and other fields that have little GIS expertise. I’ve been working recently voluntarily on some humanitarian GIS projects, and the major problem is not the server setup, but that the base data is not interoperable, nor oriented towards GIS visualization or analysis, although it would require little effort to make it so.
The most important point that Jame’s post raises is the Web 2.0 aspect– the community involvement. That is where it is at. Technically, all this is relatively kid’s stuff. The cost of software is an obstacle, but ESRI, for example, has a long tradition of supporting humanitarian/non-profit work — it gave me a free ArcGIS license for my not-for-profit avian flu and other spare time humanitarian work. Hardware is not really an issue; I’ve inexpensive host accounts that have a decent high capacity.
What’s more interesting here with the server-based approach that James suggests, and which I endorse, is the real time community aspect. Whereas I made the avian flu maps as a demo of what was possible, and why current GIS users weren’t exploiting the data they had sufficiently, or the web visualization techniques, it is nonetheless very limited.
In the server-based mode that James and I envisage, this GIS resource would most importantly be handed over to the users and producers — mainly top flu scientists. Data would be entered in real-time by a community of acrredited users, who would also have rights to edit/correct data; ie that is one brings a community of experts to contributed to a well-structured geodatabase. All would contribute to a central online database, instead of their partial non-GIS-enabled datasets.
Moreover RDF and other data structures could link out to genomic and other data, and also include fields permitting not just visualization, but proper GIS analysis.
A few national databases, such as those of Thailand, include sufficient detail to allow decent GIS analyses. Most are a disaster. Those of the international agencies in charge of health are not there yet, but increasingly they are aware of the power of GIS analyses, and the need to have adequate data to allow this
.
So my thinking is that while I’m with James on this 100%, the big problem in public health is not technical or cost, but awareness of GIS and the need for high quality data. The server/community route is clearly the one to take, and the GIS community should send out this message.
We looked at ArcGIS Server a year ago, but our customers weren’t yet ready.
Now, having seen some of it in action at the show, they are ready. There is one AGS app (.NET) being deployed here in the next month, and many more to follow.
The question will be in looking at the right performance mix (currently most of our apps are ArcIMS-based, and some apps and several components may stay ArcIMS-based), and how best to leverage the technology. Build-out of services is a no-brainer, but having those services properly architected and accessible in an ESB solution, and with BPEL and other tools in mind will be a challenge.
Other challenges (more for ESRI to consider), will be more consistency and standardization in ADFs across AGS, AIMS, Geospatial Portal Toolkit, and other offerings. We do not want to see ESRI upgrades roll out, which in turn break our existing apps.
Expect to see one or more of our apps built on AGS at next year’s conference…
James do you get the latest versions of Arc Explorer? Are you running what we say at the conference?
Ha! No I’m on the same ArcGIS Explorer Beta 2 Build 222 that everyone else is on. I’ve never gotten beta products beyond those that I get access from because of the ESRI 9.2 beta program.
I’m sitting on my hands waiting for the next release of AGX like everyone else.
“the big problem in public health is not technical or cost, but awareness of GIS and the need for high quality data.”
Being a university summer student in a Public Health organization.. and constituting the entire GIS department, I can’t stress Declan’s point enough. At the start of this summer I put together a day long seminar for all the management here because people here were actually outsourcing basic cartography work, and simple geographical analysis. They had no idea what the “Geography Inspection Student” did, or how it could be important/relevant to public health.
Oh, and sorry for double posting, I meant to address the point about the need for high quality data.
Trying to perform analysis or presenting statistics via GIS becomes incredibly difficult when there is compromised data, whether through carelessness, incompleteness or simply a lack of useable data itself. So much more could be done if everyone who partakes in data collection and organization, keeps GIS end users in mind when putting together packages.
I find this discussion intersting. I currently am working on a project contracting for the Center for Disease Control. To me, they have plenty of qualified GIS analysts, and the data for our project is of good quality. The problem I often see is a lack of knowledge about what can be done in spatial analysis regarding health. I have had to spend alot of time with our clients at the CDC explaining benificial spatial analysis and justifying how GIS could provide insights into the statistical data present.
As well, there is alot of overlap in CDC programs. The nature of funding tend to be a large source for this overlapping as well as a lack of clear defined requirements for these projects. Also, there are so many layers trying to determine who can provide and who has access to data.
Just thought I would put my three cents in.
I’m going to chime in with my support for Tim’s aforementioned points. Using PostGIS, perhaps Geoserver, and some clever scripting, you can server up large quantities spatial data at near real-time temporal increments to various clients – GE in particular. And the cost is basically free.
Not to knock ArcGIS Server, as the new one looks very capable, but I would look elsewhere for my WMS/WFS/KML streaming needs.
“Not to knock ArcGIS Server, as the new one looks very capable, but I would look elsewhere for my WMS/WFS/KML streaming needs.“
Unless of course you are locked into the ESRI suite. ArcGIS Server now allows those groups who could not run PostGIS/Geoserver/Mapserver publish data that the community can consumer, rather than just other ArcGIS clients. Lets not lose sight of the big picture here for ESRI users.
Yes, true indeed – ESRI shops will be very reluctant to move towards an open source solution, no matter the cost savings and benefits gained from moving towards open standards. However, I still believe that these days it is increasingly difficult to be “locked” into the ESRI product suite – considering the mass industry support for their own formats, the emergence of the OGC standards suite, and even ESRI’s own moves to read and write in more open formats.
But to your point, my thoughts are that the big picture for ESRI users is that they might think about slowly – as slowly as need be – think about moving some of their solutions to be PostGIS based with Mapserver or Geoserver or one of the many WMS/WFS servers acting as the streaming server. I do wonder about moving heavily analytical-based apps to use PostGIS – having only partially done this myself.
Dylan, why should ESRI users think about moving to open source? Because it exists?
It will be a cold day in hell when our IT department allows us to install open source on our windows computers. I have to work within that framework, good or bad.
“ESRI shops will be very reluctant to move towards an open source solution, no matter the cost savings and benefits gained from moving towards open standards“
If ESRI is giving these shops access to open standards what is the loss? Consumers are pushing ESRI for this support and ESRI is delivering. There is nothing wrong using ESRI software on the back end if you can server up data that can be consumed by many users independent of ArcGIS. That is a huge deal IMO and one that ESRI users need to take advantage of.
If as av8tor mentions that one is “stuck” using ArcGIS, they now can interact with the community at large with ESRI’s support of many OGC standards. Is it perfect, not by any means. But now that the door is open, ESRI users need to push it wide open and let all the other open standards in.
The end user shouldn’t worry about what server software is being used. That is for us to argue on blogs. They just need web services they can consume on the clients of their choice.
I just wanted to make a point about software costs. In my experience with large scale projects software costs don’t make up a large percentage of the total project cost. The fact that PostGIS / Geoserver / Mapserver (new acronym: PGGM) costs absolutely nothing has very little impact on the bottom line. What is far more critical is availability of required skill sets. Good SDE Admins with experience in large data volumes are tough to find, but I bet you could count the number of people with equivalent PostGIS experience on one hand. That is the real ESRI ‘lock in’. Add into the mix the new ArcSDE for workgroups, and the revised ArcServer lineup and it’s even harder to justify the move to different products. The only way any established company would change is if the other solution was technically superior… not just cheaper. As it stands right now while PostGIS is really good at maintaining simple geometry the fact is SDE is at least as good and offers a lot of other features users can grow into (images / grids / terrains / topology etc, etc). Of course if you don’t have any ESRI products & your looking at a relatively simple project a PGGM solution would make a lot of sense.
for the foreseeable future, I think Cam W. has hit the nail on the head.