Coming Up for Air
I’ve been so busy working on a 3D modeling job for the Navy that I haven’t had a second to blog. I’m usually pretty good about posting fast (if you can’t figured out, I can type faster than I can think), but these last few weeks have been tough.
I’ve been thinking about how to bring FeatureServer into workflows that require ArcGIS. Because ESRI refuses to support KML or GML complex GML out of the box, I need to think about how best to get datasets into ArcMap in a way that users can interact with it. No easy task given the requirements on my desk. We are trying to get as lightweight on the server side as we can given the budget constraints some of our clients are facing these days. FeatureServer + OpenLayers just fits that bill nicely.
I see Steve got the deCarta devZone up and running. If a banner ad with a magician can’t get people on the deCarta bandwagon, I’m not sure what can. Seriously though, if you saw the previous deCarta developer site, you can see they have something nice building there.
Well I’m sick from eating all my sons Halloween candy so I’m heading for bed. I’ve got to get some rest before fighting ArcScene and Google SketchUp tomorrow.
Oh, loneliness and leftover halloween candy are a dangerous mix.
What do you mean “ESRI refuses to support KML out of the box”? Doing a simple search on KML in ArcGIS Desktop help reveals a lot of discussion on how ESRI supports KML.
in 9.2 ArcInfo I can add KML data straight into a data frame
ESRI does not support KML or KMZ out of the box. Show me where I can drop a KML or KMZ right into ArcMap or view it with ArcCatalog.
It cannot.
I’m not sure what you mean by “support GML” since it’s a 500 page behemoth with many 1000’s of pages add-on/optional specs… it’s much like saying “support XML”, what does it mean?
Out of the box, 9.2 lets you import GML and WFS data using the “free” version of the Data Interoperability Extension. You can also export a layer to GML using the Simple Profile.
If you’re worried about GML, I’m not sure why KML is relevant to the mix? Isn’t the point of GML to get away from those evil “proprietary” formats like KML and Shapefiles (even though the specs are published and well-known)?
You are right, you can import simple GML, but support is still substandard.
As for the “why I’m worried” about excellent GML/KML support, that has to do with the specs of the scope of work. It doesn’t matter that you can import KML into ArcGlobe or simple GML. It matters that in 2007 ArcGIS is substandard on two data formats that should have been supported years ago. I mean cartographic representations are nice, but how can KML not be dropped into ArcMap at this point?
Amazing!
Ooops. beat me to it so I changed this post (i.e. only Globe can handle KML).
You’re right–the KML support is less than desirable and could be far better than it is. And if the demand is driven by user requirements, then logic has nothing to do with it.
I’m still not clear on what you’re looking for in the GML area? Supporting full GML 3 is not possible as it’s like supporting XML–there’s an infinite number of ways to express your data and no single, generic reader will do it right.
Right, which is a problem.
Am I the only one flabbergasted that we still don’t have KML support in ArcGIS?
Weird–I rewrote that comment (since I didn’t see your #4 post until AFTER I posted) and its not showing the changes. Argh.
I’ve got a GML that the client wants to be able to view.
I mean I understand your point about GML, but if OGC products can create GML that ArcGIS cannot read, how can ArcGIS be OGC complaint?
Aiee! I’m in comment heck! (now the changes show up)
I have long considered the lack of KML support to be a case of sour-apples from ESRI. No matter how they viewed the Google/Keyhole product, they knew that KML had become/was becoming a defacto standard.
Bitterness is unattractive no matter who it comes from. And all it’s done is made our lives harder.
In regards to GML–aside from WFS/WMS, I’m still not sold on the concept. This may have to do with the fact that I’ve read the standard cover-to-cover several times now and am no closer to understanding how it’s supposed to make my life easier (am I the only person who doesn’t get ‘it’?)… but everytime I’ve used the GML3 standard to deal with anything except WFS/WMS, I’ve had to use/write custom, task specific interpreters that had intimate knowledge of the xml schema/tag hierarchy… So now I can re-use my polygon-parser class, but that’s a tiny piece of my problems.
Coming up with an XML-ish way to describe spatial constructs was a cute idea, but I still have to embed those GML tags into my own, proprietary XML document that uses my own convoluted sense of “logic”… or, as the case is, forces me to understand and search someone else’s hideous XML documents to uncover a few nuggets of spatial data.
Thanks, but no thanks. I’m ready to just go back to CSV’s… oh, wait… ArcGIS doesn’t handle those out of the box either.
Not just you Cellulose:
‘Import Drawing – GML
GML is an abbreviation for “Geography Markup Language.” A spectacularly inefficient format, GML is based on XML and is being touted as a proposed standard in some circles. Unfortunately, GML is something of a non-standard since every implementation of GML to date for saving GIS data has been incompatible with other implementations. Besides the intrinsic incompatibility designed into GML, the main problem with GML is the extreme inefficiency of the format: GML will frequently require files that are over 100 megabytes in size to store GIS data that other formats can save in only five megabytes.’
I’ll let you guess which GIS program’s Help file that comes from
Am I the only one who doesn’t WANT KML support? I unfortunately don’t live in the free for all happy GIS for everybody world that some of you live in. I need data structures that are SECURE and KML doesn’t come close to that. KML is a liability where I work and usage of the format is banned (along with shapefiles). KML is just the 21st century Shapefile. It is a simpleton little format that I hope no one ever tries to use for GIS analysis.
Viewing, sure….but ArcGIS pops out KML from layer files “right out of the box”. So you can do all of your analysis using robust data structures and then water them down to KML for viewing.
I refuse to adopt any “open” data structure for the sake of “openness” that lacks so many basic functionalities that enterprises who make a living off of their data come to expect. I think ESRI sees it the same way and will support GML/KML when they mature into an format good enough to play with the big boys.
Me persoanlly, I just want it to support more formats, especailly open ones, out of the box with out making me purchase data interoperability extension. You know, like Manifold already does today. So for example, I want to do something very simple in an ArcGIS app, like map a table from a SQL server database that has a field that contains geometry for points, lines, or polygons as WKT. You can do it in manifold out of the box. You can do it in ArcGIS but you have to program your own custom layer – yuk!! What’s up with that? It should be built in like mapping an XY event table thankfully is, and yet it is not.
OMG, I knew someone would say this.
From the ESRI help:
ArcGIS supports this GML simple features profile out-of-the-box for all users.
that said -all users- You don’t need a Data Interop license to read GML out of the box. It uses Data Interop, but you are getting GML support FOR FREE.
Furthermore, to be thorough:
“ArcGIS also provides an optional suite of data conversion tools as part of the Data Interoperability Extension so that users can add custom GML formats that support any particular GML profile. Using the Data Interoperability Extension, users can work with a series of GML profiles.”
So for the complex GML stuff you DO need an Interop license.
Please please PLEASE people, RTFH before jumping on the ESRI bashwagon.
@J Wallis
What format do you use then?
What do you mean by secure? Data integrity? Encrypted storage? Signing? Chain-of-custody? ACL?
I haven’t found any format that deals with those security mechanisms very well–if at all.
im pretty sure this is not related to any extension ive installed, in arcmap 9.2 I can opne kml/kmz straight from the add data button
http://i7.photobucket.com/albums/y254/jak-c/Clipboard01.jpg
@Cellulose
Enterprise Geodatabases or Oracle Locator for enterprise datasets. If it can’t support row level ACL’d security then we don’t touch it.
@Chris C.
That help-entry alone might be enough to get me to use that other product. I’ve finally found a kindred spirit.
I’ve spent years trying to convince people to use the various binary-XML formats that have come along to try and reduce XML-induced overhead… but then I guess I like those lost causes.
On the plus side, GML can be compressed really well–down to just 2x the size of other formats.
@J Wallis: I’ve RTFM many times and it still says the same thing. Full KML and GML support is not “out of the box” as I’ve posted again and again.
I’m sick of having to find expensive solutions to problems that clients have.
@simon jackson
I’m not seeing that on my ArcInfo 9.2.
weird.
ive got it on my other computer with 9.2 arcview.
Ive installed a script which exports to kml but it would not affect this. Ill try and determine how ive got it and others havnt. Works a treat (as long as everything else is in wgs1984).
@James Fee
If your clients are hell bent on using KML/GML they need to have their heads checked.
Standards are generally draconian and slow to change. OGC and google rip out a new “standard” of their formats every 6 months it seems. How can companies (or software vendors for that matter) afford to keep up with that?
The baseline for data interchange is the GML simple features profile. ESRI supports it at 3.1.1. ESRI supports KML at the 2.0 release. If that isn’t “complex” or “full” enough then I guess you should continue to get mad at ESRI, but I think they have better things to do than support unstable data structures.
If there is one company that knows something about data interchange formats, it is ESRI. How many software packages can you think of that work with shapefiles? It is de facto. Crappy format, but de facto.
@J Wallis,
Ok–I think I understand your point and agree with the principle. KML/GML/Shapefiles/etc don’t come with the infrastructure necessary to provide enterprise-level ACL.
If you want it with KML, you have to implement it yourself (i.e. custom web apps to generate custom KML’s, etc). And even then, it’s a kludge.
Though, I still think RDMS solutions still have a long ways to go before I would call them “secure.” How about giving me ENCRYPTION over the network by DEFAULT, instead of forcing me to muck with the “Advanced Security” tool or whatever Oracle calls it this release. How about giving me some control or integrity of the data once it’s been sent to the client?
adopting encryption at the database level is crap anyways. It needs to be done at the SAN/NAS level, handled by a dedicated controller on the network. That gives you a uniform encryption plan for all data, not just your RDBMS.
@J Wallis: ESRI doesn’t support KML 2.0 out of the box so that argument is bull. Buying an extension to read KML is a major PITA for interoperability.
As for the clients need for GML/KML support, they have good reasons for needing it. What it will come to in the end is their need for GML/KML support will trump the need for ArcGIS and in the end that will be a huge loss for ESRI.
http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?id=2939&pid=2932&topicname=KML_in_ArcGIS
That isn’t KML support out of the box? I give up.
J Wallis:
Standards are generally draconian and slow to change. OGC and google rip out a new “standard” of their formats every 6 months it seems. How can companies (or software vendors for that matter) afford to keep up with that?
KML? The standard doesn’t “change” every 6 months, but even if it did ESRI doesn’t support it by default. They have their head in the sand here.
The industry is moving forward and ArcGIS is like a big anchor keeping folks tied down. By the time they free themselves, they’ll be irrelevant.
@J Wallis
ArcGlobe isn’t free. At least not in my neck of the woods.
@J Wallis
I think we’re talking about two different things.
SAN/NAS encryption (i.e. encryption of the physical database files), while a very important problem, really has nothing to do with the client/server security problem I’m worried about.
It also doesn’t impact data integrity at the application level. The data may be encrypted on disk via my NAS/SANS, but it doesn’t guarantee that someone with access to the server (i.e. after the encryption/decryption phase) hasn’t tampered with my data…
And it sure as hell doesn’t give me any capabilities to discourage my users from abusing the data once it reaches their client machine.
@James, J Wallias
And getting it into ArcGlobe doesn’t mean you can get it into ArcMap or ArcCatalog.
last time I checked, no ArcGIS product was free. Are you bemoaning cost or functionality? The functionality is there, but yes, in order to make money sometimes you have to spend money.
I can’t tell you how many times I have worked with companies that wanted to “rage against the machine” and not simply adopt an existing solution. 9 times out of 10 they ended up spending more money than if they would have simply bought the (usually ESRI) product. But hey, as I used to say (and still refer back to) “Consultants: If you aren’t part of the solution there is good money to be made prolonging the problem”
And Lefty, if you call data formats tied to a single geodetic reference “moving forward” then I would hate to see what your example of moving backwards is.
J Wallis – yeah I need it in ArcGIS Engine Runtime though – not desktop (back on connecting to a layer that is a table in a SQL Server containing WKT w/o having to write a custom layer). If you could point me at some help for doing that I would be grateful. Hopefully saying that won’t make your head explode in rage.
@Cellulose
there is no encryption solution that can solve data abuse once it reaches the authenticated client’s desk. A bad person is a bad person, and you can’t simply reboot them and make them start working ethically. You can’t solve basic HR problems with technology. If you have an errant user, you establish consequences for not following policy.
There are very good reasons to buy 3D Analyst. To read KML into ArcGlobe is very low on that list. But that doesn’t change my original statement that said you can’t read KML in ArcGIS out of the box.
As for spending money on an “ESRI solution”, that isn’t possible as the way the Data Interop extension handles KML/GML renders it useless. It has nothing to do with my comments on the scope of work on my desk. I’m guessing that it will be a server side ogr2ogr solution, but that is yet another step without ESRI.
ok, it is time to split some hairs here and get the terminology/product names right.
You can’t load a KML file into ArcMap, which is part of ArcGIS Desktop. You can load a KML into ArcGlobe, which is also part of ArcGIS Desktop.
The statement “ArcGIS doesn’t support KML/GML out of the box” is incorrect since ArcGIS is a suite of applications and of which certain applications of that suite do accept KML as a client format.
Furthermore, the statement “ESRI refuses to support KML” is wrong on face value but subject to interpretation if you want to view that statement as an opinion. They may seemingly refuse to support it at the level you want (KML 2.1 2.2, or whatever flavor Google is pushing these days), but that still doesn’t omit the fact that ESRI supports -a- version of the KML standard.
We are in the weeds here, but I think James’ wording is correct. Buying add ons to a product does not mean it supports KML out of the box. I think if he said ArcGIS didn’t support KML and left it that way, you’d be correct. But he did add that “out of the box” to the end.
When I saw the comment count I thought you’d been hit by a spambot. lol
I can’t comment intelligently upon KML support because I’m a newb and haven’t fooled with it. With sharks like Google and Microsoft swimming in the water though the last thing you want is to appear overpriced and unresponsive to customer needs.
ESRI is responsive to customer needs. That’s why KMl 2.0 IS supported “out of the box” by the ArcGIS suite. So it isn’t supported in the ArcMap 2D application “out of the box”. boo hoo. Google doesn’t have a 2D desktop product to support it with either. But both companies support it in a globe viewer product. And both companies support it in web applications.
Microsoft is going to buy ESRI one of these days anyhow when Jack decides to retire, so ESRI doesn’t have to worry -too- much about being unresponsive. They just need to be responsive to the right people.
And here I thought there were 37 comments congratulating me on the new devZone…
@J Wallis
Technology can provide some vague level of control that can prevent casual abuse (i.e. unintentional or novice level abuse). Adobe Acrobat, Sharepoint, and other technologies provide such mechanisms. They are not perfect, but they do help to enforce policy and discourage casual abuse.
Perhaps my point has been obfuscated. Database-based GIS formats may fix one security problem (ACL), but they still leave so many other problems that calling “secure” is misleading at best.
“ESRI is responsive to customer needs. “
Only if you spend a lot of money. ESRI has proven year in and year out, those with money to spend get the features, no matter how unimportant they are to the community at large.
KML import/export support from ArcMap is key to the future. I’m not saying it has to be as complex as A2E, but dammit we deserve to be treated better than we have.
I’m sorry J Wallis, but boo hoo? That is the kind of reaction that has gotten all these new programmers out there doing Web GIS using open source and other solutions and not ESRI products.
Boo hoo is what I’ll be saying in 5 years when ESRI is bought by Pitney Bowes.
“I’m sick of having to find expensive solutions to problems that clients have.”
Why don’t you try Manifold?
yeah, I don’t understand what the problem is. A $100 runtime version of Manifold and about 6 lines of VBScript, and you are done.
Hell, I’ll write it for you:
DrawingName = InputBox(”Enter Drawing Name”)
Set theDrawing = Document.ComponentSet(DrawingName)
Set KMLExp = Application.NewExport(”KML”)
KMLExp.Export(theDrawing,”c:\temp\”&DrawingName&”.kml”,false)
There, you are done!
I don’t need Manifold to do that. I can do it already for free.
I thought we were talking about your clients?
@Erin
“I’m sorry J Wallis, but boo hoo? That is the kind of reaction that has gotten all these new programmers out there doing Web GIS using open source and other solutions and not ESRI products.”
And there enlies the problem. I’ve had many a programmer come running to me, “ohh ohh, look what I can do with KML and GE!” Then I ask them to provide me the data in something other than WGS1984 and serve up features only to appropriate authenticated users. Usually all I ever get is blank stares. Sometimes I get the “well why would you want to do that?”
Hence my point of the overblown need for KML interchange. The thought of enterprises using KML as their corporate data format scares me…but if I were their competitor it would absolutely delight me.
And on another thought, who is actually doing GIS analysis with KML/GML? I mean the hard core type of analysis we’ve come to expect with ArcGIS, not some simple buffering or clipping. I’d love to see some robust examples.
@Mike Summer: We are and they want to use ArcGIS Desktop with the KML/GML generated.
I can use ogr2org to convert to shapefiles, but that adds a step. Manifold adds nothing to this problem irrespective of the fact that it can handle the conversion.
@J Wallis: The issue is that the GIS Analysts want to get the KML into ArcGIS so they CAN do analysis. That is why KML import is important.
@J Wallis:
With respect to secure access to data: that is what SSH was developed for. Why windows does not come with this (free) service is beyond me. Try using a real operating system some time.
As for KML and GIS analysis. No one is pushing analysis within the context of KML or GE. Furthermore no one is pushing standardizing around GCS/WGS84.
A similar example can be made of shapefiles. Take a look at vector data which has been created in the last 5-7 years, and you will find a pile of topologically incorrect spaghetti. How about all of the fun associated with storing attributes in DBF files?
James is asking for basic functionality which open source software can provide- and the (how much is that ArcInfo license these days?) “leading” GIS app can’t. What is the big deal?
J Wallis:
I’ve never touched an ESRI product, and all my analysis experience is in http://crschmidt.net/mapping/wpserverdemo/ , which apparently (despite doing a number of things that ESRI software doesn’t let you do without paying extra for more licenses) isn’t enough.
(The list, for the record is:
dissolve, union, difference, intersection, symmetric difference, line simplification, convex hull, centroid, and buffering.)
So, what *is* ‘real’ analysis?
@Dylan
SSH is not going to secure FCs down to the feature (row) level. And the infrastructure that you would have to reinvent to do something close would be ridiculously expensive when you could just use existing technologies. Securing GIS data is not an all or nothing scenario….unless you feel like splicing your datasets into subsets based on user.
And my point was not on WGS1984 standardization, it was that KML is stuck in one geodetic reference.
I think it’s fine that J Wallis is happy with his ESRI stack and has no interest in KML/GML support.
What mystifies me is why he feels the need to vehemently oppose the addition of better support for those formats into the products. Just because the capability is available doesn’t mean you have to use it.
I do a lot of work where data security is an issue and I probably wouldn’t choose KML or GML in that case. But I also wouldn’t choose shapefiles, SDTS, DXF or a host of other formats that are already supported.
James wants KML and J Wallis doesn’t. Unless they’re sitting side-by-side in the same shop, I don’t see the problem.
And, for the record, I agree that better support for KML/KMZ/GML is long overdue and would love to see it added to ArcMap and ArcCatalog instead of just ArcGlobe.
I just still take issue with the original comment that “ESRI refuses to support KML” because they obviously support it in some fashion. If they were truely refusing to support it then there would be -zero- ability to use KML in ArcGIS.
I’m sorry but SOA is going to keep pushing the need for interoperability. If ESRI views that as an overblown need then they’ll eventually be at a competitive disadvantage to someone more willing to keep up with the industry.
It isn’t so much that enterprises want to use KML as their standard data format. Its that they would like the ability to leverage other data sources regardless of the format and without having to spend an arm and a leg for something that should come in the box.
ESRI doesn’t see it as overblown, but I guess their customers do since ESRI isn’t working overtime to build such support “out of the box”.
Ironic that people clamor for this stuff “out of the box” when there is good money to be made by providing a well written ArcGIS extension. It sounds like ESRI is providing a good opportunity for someone to make money to provide KML capability to the 2D application side of ArcGIS.
I know if KML support “out of the box” was the only thing I had to complain about with ArcGIS my life would be much easier.
That’s something I think we can all agree with!
Sure, we can all have a bitch post were we list the millions of things that bother us about ArcGIS, but this post was about a client who wants their GIS Analysts to have access to KML/GML generated by another product they use/plan to use.
Holy @#$% …the old gaurd vs. the Web 2.0 crowd. I LOVE IT!
James maybe you should add a new blog posting titled: “GIS is dead…swallowed by IT”…and see what kind of response you get.
@simon jackson: Any insight into what may have allowed you to see into the kml from ArcMap?
On another note, I can’t see any kml listed in ArcGlobe either…
Hi. This isn’t flamebait. Really, it isn’t. But, having only used Manifold, that is my only perspective.
Can’t you just ask ESRI to include something? And, won’t they do it in the next release? Manifold typically incorporates user requests rather quickly – especially if it is some kind of one-off thing like importing and exporting KML.
So, rather than complaining here, why not just ask ESRI to do it. Especially James, because you are a business partner. If they won’t listen to you, then maybe you have to re-evaluate how committed ESRI really is to their user base. Maybe they are only trying to protect their economic turf – which doesn’t bode well for the user community.
“complaining” can be called healthy discussion. ESRI does implement lots of enhancement requests – even more than Manifold, becuase the products are bigger and broader! ESRI has even implemented some of my feature requests, where-as manifold has not. That’s not to say one is better than the other, it probably just means that more people asked for the same thing as me for some of my ArcGIS requests than for some manifold requests. There are proper channels at both companies for enhancement requests, and like all software companies not all enhancement requests (e.g. candidates) can be implemented – so they weigh against frequency of the request and strategic product vision and direction. So that means that if you really want something, your best bet for getting it is to log the enhancement request via the proper channel with the software vendor, but then ALSO to advocate for it in user forums – which gets other users who see it and want it to log the same request, and also exposes discussion to the vendors both of whom lurk here – so when they see a hot topic or things people are freaking out over in blogs, they can use that in their decision making process when picking feature candidates to implement.
@NoLove
You see where I’m coming from…. Google shouldn’t be discounted for not having a desktop application for the same reason it shouldn’t be assumed that Microsoft will buy ESRI. You can bet both companies have people who are attempting to come up with a view of GIS five years down the road and whatever the shape when you’re dealing with uncertainty the solution has to be flexible.
Also, when you have Google and Microsoft interested in location-based services how long does it take before IBM’s interest is piqued? If the GIS of the future is a web 2.0 application sitting on top of an analytical API going against a spatially-enabled database then who is in the best position to capture market share?
@No Love
I think it is more fundemental than “old guard vs Web 2.0″. It is “flash web programmers vs geographers.”
This is probably meant for a totally different thread, but someone should start tallying up the costs associated with having programmers with no geography skills implement GIS. Working with data formats that have geodetic limitations are only the tip of the iceberg.
@JWallis
I have no time for flash but I do have time for AJAX. We would not be having this discussion if it were not for non-GIS programmers. KML, Virtual Globe technology, and Web 2.0 are all thanks in part to computer software engineers and not GIS programmers.
I come from a more traditional geography/GIS background, and I would rather hire computer programmers and train them on GIS then hire GIS analysts and train them how to program. Now there is some waste!
There is a place for geographers and GIS staff, but let’s face it GIS is not as complicated as we all (the GIS crowd) tend to think.
Long gone are the days of AML…at least I hope.
@No Love
send some of those magical programmers my way then.
@JWallis
I think they have been talking to you for the last 60 or so comments (minus yours).
Being a web programmer myself (not sure if I’m “magical”), I can honestly say that this discussion brings to mind that arguing that KML is unfit for GIS Analysis is like arguing that AJAX is unfit for programmers. You can argue all you want, but in the end the benefit of Web 2.0 technology isn’t for the brainiacs behind the scenes — it’s for the benefit of the USER experience. KML has been (AJAX, too) an integral part of popularizing GIS or location-based applications, if you prefer. ESRI needs to (and I believe they know this) keep up with the times if they want to survive. They’re already falling way behind in the Web-based map viewer race. I’ve already converted several burgeoning ArcGIS Server sites to Virtual Earth.
I agree with No Love. GIS is simple! The difficult part is the development of a ” web 2.0 application sitting on top of an analytical API going against a spatially-enabled database “. Look at power users of GIS…I am one and I have no formal training in it. i just started up Arc what ever and Manifold and plugged away until I learned what I was doing (about two days). I try to do the same with progaming (years) and I still have just begun.
I’ve been programming for a long time, most of it in the GIS world. I don’t know that I’m prepared to say that GIS is “simple.” If you find it to be simple, then there’s a lot more to discover.
I would suggest that you not forget that all of those APIs out there wrap some pretty complex operations that people with an advanced understanding of both geography and programming packaged up to make it look “simple.”
@oakfish–
Thank you for being sensible.
As for this latest version of “what starts as an ESRI feature commentary morphs into a ‘Whither GIS?’ debate”, I will only add that in 18 months the majority of location-based analysis will be done by .NET programmers using SQL statements against spatially-enabled databases –with the visual output being piped to a “lightweight” web interface.
Whether you call it “GIS” or not is irrelevant. Whether you turn up your nose at “buffers & intersections” is irrelevant. This is where the explosive growth is going to happen. Should you choose not to participate, fine. But the next generation of developers will not only be liberated from the burden of these endless desktop debates but also blissfully unaware of the misplaced condescension of the GIS priesthood.
Brian
I hate that this sounds like a lovefest between me and Brian, but dude — that comment says so much so well. Well said, indeed.
Well said Brian!!
What an interesting thread….
Let’s start with the most amazing fallacy:
“I will only add that in 18 months the majority of location-based analysis will be done by .NET programmers using SQL statements against spatially-enabled databases”
Brian, I respectfully disagree. That is about as likely as people giving up Excel in order to peruse “spreadsheet analysis done by .NET programmers using SQL statements against data-enabled databases.”
The number of people who prefer an entirely visual, point-and-click GUI within a modern GIS to any sort of SQL (let alone any sort of .NET programming) outweighs the text interface , “let’s all code our own server-side appliance” brigade by at least 1000 to 1. That’s true of working with spatial data regardless of where it resides, be it in file-based storage or fetched in seamless connections to a spatial DBMS.
In fact, that’s especially true of spatial DBMS storage, as anyone who has ever tried to market a spatial DBMS (like Oracle Spatial) will tell you. The first thing they all realize they need once they engage the market is a highly visual interface that eliminates the need to write code to do routine analysis. Otherwise, they find themselves at the mercy of whatever rich GIS interface customers choose to consume and manipulate data stored in the spatial DBMS. Listen to ESRI brag about how they strong-armed Oracle with Arc-middleware and you’ll know what I mean.
Sure, there are developers who will create server-side applications in the style you discuss. Those applications will be outnumbered by about 1000 to 1 by people who simply use the spatial DBMS for storage and do all their work through point-and-click analytics within a modern, visual GUI like that in Manifold.
Regarding KML:
It doesn’t have to be the best thing on the planet or the standard to end all standards to be worth supporting. It simply has to be something that is used by 100 million Google Earth users. That makes it by far the most widely used (perhaps by a factor of 100 over whatever is in second place) spatial data standard around in terms of the number of people for whom it is their native spatial data format. It’s a trivial thing, anyway, to support, so may as well do it.
Turning to James’ comment:
“I can use ogr2org to convert to shapefiles, but that adds a step. Manifold adds nothing to this problem irrespective of the fact that it can handle the conversion.”
On the contrary, it adds a lot… James, for shame: as an experienced GIS guy you know it is almost never the case that one should blindly convert one format into another and fire it off (although of course as experienced GIS guys we all are especially appreciative when we can get away with that…
Realistically, you want to take a look at the data after import and before export so you can fix the usual, hare-brained errors in matters like datums, coordinate system parameters, etc. that have been inflicted upon you by whoever gave you the data, assuring you it was good to go as is. A real GIS like Manifold allows you to see whatever you’ve imported in context with however many other layers you want, so errors like wrong coordinate system parameters leap out as data is visually misaligned (here we have an example of why the whole world prefers visual GUIs to black box operations with .NET programs that call SQL statements).
A real GIS also enables you to do the usual slicing and dicing you might want to do with data as well. Suppose you get an AutoCAD dxf showing parcel boundaries as lines from that friendly CAD guy in the streets department. You know, the guy who hands it to you proudly calling it a “map” and gets a blank look when you ask him anything about projections or georeferencing.
You’re going to want to georeference that .dxf, clean up the usual CAD errors (dangles, overshoots, boundary lines not closed, etc.), create area objects (”polygons” in ESRI-speak), and so on *before* you put your name on the work product by exporting it as KML or anything else. A real GIS like Manifold lets you do all that within the comfort of a vast array of supportive tools and visual constructs that save you time, reduce errors and ease you on your way to a blissful weekend. [Don't get that either with a simple format converter or from the .NET guys writing SQL code who will be doing everyone's job for them in 18 months...
]
And, given the interest in spatial DBMS, you’re going to especially want to be careful with the data by giving a good examination and cleaning before you upload it into your spatial DBMS. Don’t want to contaminate that central data warehouse with bogus data!
So, yeah, sure it makes sense to use the real GIS of your choice to do format conversions, and that’s a good argument for your GIS being able to read formats of interest like KML.
Can’t resist nitpicking one comment, which I realize was offered in good faith so I admit in advance this is nitpicking:
“ESRI does implement lots of enhancement requests – even more than Manifold, becuase the products are bigger and broader! “
I’d respectfully disagree with the “more than Manifold” bit. First, count up the number of new features released every year and Manifold is far ahead. Many of those are responses to user requests. Second, look at the elapsed time from when users start to ask for something and it gets done, especially noting really significant requests that users especially must have to stay competitive – Manifold does those much faster.
Examples that come to mind are support for Vista, native 64-bit code for all products, bona-fide use of multi-core processors, integrated IMS, direct usage of native spatial DBMS [Oracle, DB2, Katmai, PostgreSQL] and built-in use of image servers. I could be wrong about this, but as far as I know ESRI is far behind in those areas despite a lot of user requests. ESRI probably will do most of these things, but years after Manifold.
Third, I’d respectfully disagree that ESRI’s products are bigger and broader. No ESRI product comes close to Manifold Universal in breadth or depth. Not even close. At best, you have to glue together [whatever ESRI is calling them now...] ArcInfo ArcSDE ArcIMS ArcObjects parts of 3D Analyst. And even with that ensemble all put together I don’t think ESRI is doing 500 to 1000 new features a year as Manifold does, nor are they doing really major technological things like using NVIDIA CUDA to get supercomputer speed on a desktop for a few hundred bucks.
Regards to all!
Cheers to Brian for adding some sanity to this thread.
Hey Steve,
Congrats on that devZone, Dude!
Good thread. Not much more that I could add except a couple of anecdotal comments.
1) Our own hiring practice is heavily weighted towards competent programmers before GIS experience. Given our experience in building sensors and creating mapping products perhaps we are tilted towards that which we lack. However, it was far easy for us to give GDAL to our programmers and explain some of the geospatial terms and meanings than it was to give GDAL to those with GIS experience and expect them to write the code to process near-real time reprojections for terabyte size files.
2) Our previous government clients have got the KML-jones bad, including the Navy. It appears that Google has put the hard sell on them to use Google Earth Enterprise Services. This is reaching into the USACE and other branches of the DoD and civilian agencies. Anecdotal evidence, but I think it points to some of the push we are seeing to incorporate KML as part of an interoperability standard.
3) Betamax was a better standard than VHS. Is there a useful analogy here?
@Paul
pushing KML works great for the military since they are standardized on WGS84. That is where the format advantages end though (if that even were one). However the Google Earth enterprise product doesn’t even begin to address the analysis needs that the military requires.
Doesn’t Google keep track of the usage of their product? Like, viewing or importing data.
Why would the military want/allow a third party know what data they are viewing, what type of analysis being done, so on and so forth? Even if it’s not classified work. Wouldn’t one think it’s not wise to have a third party looking over ones shoulder or even maintain a listing of what has been done.
It’s my understanding why some use WorldWind over GE because of those exact concerns.
KoS
I -think- GE Enterprise is, or can be, a self-contained solution. On the server side, you’re connecting to your own internal GE Server, so there’s no need to connect to Google’s GE servers where they can monitor your activity. On the client side, I would assume the military has firewalls that can block any attempt by GE desktop to transmit usage information.
GE Enterprise doesn’t offer the advanced analysis that an organization like the Navy requires, and nor should they. The best option would be for GE Enterprise to focus on being the best presentation layer possible while interfacing with existing systems that have been created to do the analysis. This seems to be the future of GIS – a separation between products offering presentation of data versus those that offer geoprocessing and analysis.
James, thanks for the FeatureServer link. I will give it a look. I’ve been using MapServer/Mapscript in a similar fashion – as a RESTful service under ASP.NET. Mapscript makes it possible to point to existing .map files or dynamically create maps in response to user requests. You can work with existing data or dynamically create “virtual” layers as necessary and return results as images, GeoRSS, json, VE pushpins, or whatever your heart desires.
I seem to have problems editing my comments. I wanted to add that in ESRI’s latest newletter they mentioned their own “Aerial Imagery” appliance that reminded me of GE Enterprise. Instead of using the ESRI servers, you would purchase this appliance and point your ArcGIS Explorer to it instead of ESRI’s public servers.
@KoS
Google earth is like crack….once they get people hooked on it they overlook things like privacy and security. Never in my life have I seen people fall over themselves for such a weak app.
But if it was called Microsoft Earth you can guarantee people would be all over the usage tracking issue.
And yes ESRI has an appliance that you pop in behind your intranet. Unlike google it isn’t tied to one app. You can point AGS at it and serve it up as WMS. I haven’t heard the cost but I’m strongly considering it because I don’t want my internet connection saturated going out to their services.
I assume Microsoft monitors usage. After all, their licensing for VE is supposed to be “X” number of tiles per day free, then you start paying. No complaints from me.
“Third, I’d respectfully disagree that ESRI’s products are bigger and broader. No ESRI product comes close to Manifold Universal in breadth or depth. Not even close”
I like Manifold a lot. By broader and deeper I mean there are some capabilities in the ESRI stable not in the Manifold stable, to name a few:
- ability to do GIS in the “cloud” – meaning the Web API’s of ArcWeb Services (like google maps API, MS VE APIs, and the rest) – I am not aware of Manifold having anything similar?
- mobile apps designed to run on mobile operating systems – meaning ArcPad, ArcGIS Server Mobile ADF, StreetMap Mobile SDK with 2.5d maps
- cross platform support, only one example -ArccGIS Engine QT controls for cross-platform development – as I understand it Manifold runtime engine is for Windows only?
- sheer size of ArcObjects (thousands of interfaces) – like for example I looked up MapServer in Manifold help and it has around 30 properties, around 20 methods, and is covered on a single page in the help. The Manifold scripting reference shows something like 270 objects. I think it seems smaller.
- deeper like GIS server is scalable beyond one single server out of the box (though I admit the price does not scale well at all), as well as the larger object model providing way more fine grained objects
- free geobrowser + SDK – ArcGIS Explorer – similar to Google Earth, but with geospatial analytical capabilities + robust SDK, I don’t think Manifold has released a free geobrowser
I understand anyone can think ArcObjects is too big, and indeed its difficult to find your way round sometimes, and parts of it cost a lot, and it stinks when you bump into something you think should be in the base product that is in an extension, and even that sometimes less is more. I understand the mfold business model is to not develop for non-windows platforms or mobile operating systems, and I think their model is elegant and streamlined – which definitley allows them to be very quick and agile by focusing only on one main product – and they have released some wonderful innovations. I like Manifold a lot.
I was just responding to I think misleading comments suggesting ESRI is sitting around not releasing anything new from version to version and SP to SP, because its just not true. Because of the breadth and depth of their line, add it up and they release lots of stuff each year spread across it all. They have been very busy latley, just like everybody else.
Worst jobs for the 21st Century
“Like computer programmers. Despite all the advances — and expected job growth — in the computer industry, expect the number of programmers to increase by about 2 percent between 2004-2014. Why? Outsourcing.”
http://www.msnbc.msn.com/id/21311274/
I have no idea what the hardware costs are for the ArcGIS Data Appliances, but the data costs start at around $15k for data. Obviously, it can be way up or a bit down depending on your needs, age of data, etc. I had an option to serve it through ArcIMS, ArcGIS Server, or a data appliance.
We have a worldwide dataset served from within our ArcGIS Server infrastructure in-house. Completely self contained, no phoning home, or unwanted activity logs… just the ones I keep.
The same dataset from Google was around $75k and I would have been *forced* to buy the GE Server appliance at a cost of 6-figures. The GE appliance was self-contained and could operate without phoning home… though they were evasive when asked how to sanitize the appliances of usage information.
– sheer size of ArcObjects (thousands of interfaces) – like for example I looked up MapServer in Manifold help and it has around 30 properties, around 20 methods, and is covered on a single page in the help.
Nice post. I won’t address everything, but here is one example of where you may be confused. In Manifold IMS, if you grab the MapServer, you can get the document, or application object:
Set ManApp = MapServer.Application
Set ManDoc = MapServer.Document
once you do that, EVERY SINGLE Manifold object is now available to you over the Internet – including the User Interface controls. So, all the vector, raster, and table operations are now available to you. That is way more than ArcIMS, and probably comes close to rivaling ArcGIS Server
But, you are right, ArcObjects is certainly more “granular” than Manifold, and has many more objects exposed.
“In Manifold IMS, if you grab the MapServer, you can get the document, or application object”
- thanks, you are right. I a big fan of Manifold IMS, and there are definitley things I can do with it that I need ArcGIS Server enterprise advanced for.
Following up on kj’s post with yet another book-length missive…
“By broader and deeper I mean there are some capabilities in the ESRI stable not in the Manifold stable, to name a few:”
I 100% agree with you that in any comparison of products that each have thousands of pages of documented features it will always be the case that there will be features in one product family that are not in the other. But I respectfully point out that the presence of some exceptions on either side does not a “broader and deeper” case make – to do that, you really have to consider the overall set of features. There, I still would contend ESRI has no product (and, for that matter, not even a product ensemble) that even comes close to the breadth and depth of Manifold Universal. Even though there are parts of ESRI’s product family that have features Manifold does not (as expected), so does Manifold have many capabilities ESRI products do not. And the overall balance is way more on the Manifold side of the ledger.
You raise some interesting points that merit further conversation – I hope folks understand the following is in the spirit of detailed conversation offered in respect of the good faith points raised. It’s too easy sometimes to give the impression of tit for tat when what is intended is a spirited discussion prompted by intelligent points raised.
“- ability to do GIS in the “cloud” – meaning the Web API’s of ArcWeb Services (like google maps API, MS VE APIs, and the rest) – I am not aware of Manifold having anything similar?”
Sure it does. There is the vast array of capabilities enabled through the Manifold API via IMS, which include things exactly like image servers (like GE), WMS servers, WFS-T servers and so on.
“- mobile apps designed to run on mobile operating systems – meaning ArcPad, ArcGIS Server Mobile ADF, StreetMap Mobile SDK with 2.5d maps ”
That’s a negative for ESRI and a sucker trap for developers, not a positive. It’s a bit like being proud you have an API for 8-track cartridge tapes and cassette Walkmen in an era of iPods.
Manifold feels that there are two approaches that make sense for mobile devices: 1) Web 2.0 stuff using ultra-thin browsers in the mobile device and full-power web applications running server-side on Manifold IMS, or, for those vanishingly small number of users outside the cell cloud, 2) Running real Windows with non-lobotomized applications on UMPCs.
The first is obvious and should have a lot of appeal given your first point. The second bears some discussion, since at times people forget to reckon the high price they pay for obsolete software strategies that are led around by the nose by inadequate appreciation of hardware evolution.
I’m an ex-Intel guy, like many of the folks at Manifold, and we all have immense faith, acquired through direct experience, of the operation of Moore’s law. It takes a lot of time to lobotomize something like ArcInfo or Manifold into a stripped-down version that runs within a limited PDA operating system. In fact, it take so much time that by the time you are done the PDA has evolved into a real PC that is even slimmer than what you started with, has a much bigger screen and more capabilities.
The stupid thing to do is to burn a lot of resources creating a lobotomized PDA version. That wastes engineering time and creates yet another Arc-thingy that is incompatible with all your other Arc-thingies. Your PDA Arc-thingy will forever lag behind whatever progress your company can muster in the other Arc-thingies, and it will always cost disproportionately more to sell and to support because it is a special case lobotomy product. If you are a third party developer, the last thing you should consider is developing to some lobotomized SDK because that simply guarantees that all your hard effort will result in a product that is even further removed from modern capabilities.
Manifold won’t do anything that dumb. Our standard, off the shelf product runs great in standard, off the shelf UMPCs, which are continually getting smaller and smaller. Everythying we do, every new capability we add, every feature we fix, every optimization we do runs perfectly, without modification in new genration handhelds. You can buy the standard product and take advantage of huge economy of scale, know you are getting the latest and best technology all the time, and leverage dramatically better technology for end users: much higher resolution screens that enable more sophisticated and more user-friendly interfaces, etc. That’s real depth and breadth, as compared to lobotomized obsolescence.
“- cross platform support, only one example -ArccGIS Engine QT controls for cross-platform development – as I understand it Manifold runtime engine is for Windows only?”
“only” ?
The old joke is that when the Manifold sales guy is asked what operating systems Manifold supports he or she cheerfully replies “All of them! … Windows Vista, Windows Server 2003, Windows XP….” That is, indeed the reality as Manifold really does have an exclusive focus on Windows. But that is an example of Manifold’s better breadth and depth, not a counter-example.
Manifold’s focus on Windows is not because of any personal animosity to other operating systems, it is simply catering to our customer base. Windows users want their vendors to have a focus on Windows. Windows users are fed up with vendors who blather about other operating systems while failing to do as good a job in Windows as vendors, like Manifold, who focus on Windows. In Manifold’s case, customers know that they can count upon us for fanatic and expert support of Windows with no siphoning of attention or resources to rescue people who failed to move to Windows when our customers did.
They don’t get that from ESRI, which is a black mark against ESRI’s breadth and depth for those 96% of c0mputer users who run Windows. A good example is that Manifold supported Vista from Microsoft’s earliest interal betas and today supports all Vista editions, including 64-bit native Manifold code for Vista x64, while ESRI is still blathering about maybe supporting Vista real soon some day. That ESRI takes years to figure out they should be supporting Windows standards, long after the Windows ecosystem has moved to those standards, is a huge black mark against ESRI.
“- deeper like GIS server is scalable beyond one single server out of the box (though I admit the price does not scale well at all), as well as the larger object model providing way more fine grained objects”
I defer to tom riddle’s comments about objects. Regarding scalability, Manifold is far more scalable in Enterprise applications. ESRI does not even come close. There are several dimensions of Manifold’s greater scalability ranging from simple to sophisticated.
The simplest example of scalability is also the most brutal gain in Manifold performance over ESRI: Manifold runs as true, 64-bit code and utilizes multicore processors while ESRI does not. That is a brutal and decisive advantage in scalability for over 99% of IMS and Enterprise applications ranging from the single consultant hosting web sites to some very large companies.
By being limited to 32-bits ESRI is condemned to grossly inefficient, small memory spaces. By failing to utilize multicore processors they are condemned to add physical computers to attempt to keep up with Manifold. Consider an example case:
With Manifold, you can deploy a motherboard with two processor sockets and plug a quad-core processor into each for a total of eight, 64-bit processor cores. Install 16 to 32 GB of RAM and run x64 Windows. Your Manifold IMS will eat alive ESRI IMS even if you deploy that ESRI IMS on multiple machines.
Better still, you’ll get the huge savings of running just a single server with one copy of Manifold (a mere $225 for x64 Universal), one operating system license, one license for all sorts of accessory software, one system to maintain, to power, to find rack space for, to cool, etc.
Let’s take a more sophisticated look at scaling. The real bandwidth in almost all web applications comes in the DBMS side of the application – it is the DBMS that really controls the scalability of your application.
With ESRI, you have to use whatever they are calling their bizarrely obsolete ArcSDE middleware these day. Lame! Costly! Slow!
With Manifold, you can connect directly to whatever rocket-hot spatial DBMS you choose to use: Oracle, DB2/spatial, Microsoft SQL Server 2008 spatial, even PostgreSQL/PostGIS. Without any ESRI-style middleware nonsense, you get to immediately take advantage of the endless scalability your DBMS vendor has spend billions of dollars to provide for you. Fast! Cool! Inexpensive!
And… keep in mind that your connecti0n to those databases, because Manifold automatically takes advantage of multiple processors for such things, will indeed be automatically multi-threaded through all those cores you have running. And it will be a 64-bit connection and you get all that built into that same $225 price per server, with no extra costs for Arc-middleware. And, you get the benefit of interoperability with all the other applications your enterprise employs with that DBMS because Manifold uses the vendor’s native geometry types and not some hare-brained proprietarization as does ESRI. Interoperability is a key part of scalability.
I guess the last part of the puzzle is that some web applications will indeed, even with multicore servers, require multiple servers. No problem there with Manfiold as well. Centralize your data within a spatial DBMS and any web guy worth his salt knows how to plug in more IIS servers running IMS to provide front ends. I grant you that it will require some thought in the writing of the web application, but such applications require thought anyway and scaling them out to many servers is not a significant part of the percentage of thought required overall.
“- free geobrowser SDK – ArcGIS Explorer – similar to Google Earth, but with geospatial analytical capabilities robust SDK, I don’t think Manifold has released a free geobrowser”
There’s the free Manifold Toolbar for IE, which enables any IE user to take advantage of any image server interface module from within IE: use Virtual Earth, Google Earth, Yahoo, etc.
Our view is that the world does not need yet more proprietary browsers. We already have IE and Firefox and that if the world is really interested in browsing geographic content on web sites, use IE or Firefox.
Web sites already provide all sorts of content. They provide images, audio and video using all sorts of different formats. There are endless new forms of content appearing supported by endless new types of plug-ins of various kinds for IE and Firefox. But no one in their right minds suggests, in the usual course of events, that just because someone invents a new type of content you should suddenly have to drop using Firefox, or Opera, or Mozilla or IE to browse a web site.
People only try that when they are trying to trick you into supporting their proprietary new browser. It is a time honored strategy of cornering the public Internet – sucker people into using a new browser that only you control and that is designed to herd users into the vendor’s own proprietary standards.
AOL tried that by creating an AOL web-within-the-web, exploiting the unsophistication of AOL subscribers to put a leash on them in the form of dedicated AOL software.
Since then, every web entrepreneur whose Ponzi scheme has freed up a few billion has tried to figure out how to fence off his own web-within-the-web by introducing his own browser as a way of putting a leash on your neck. And the smart entrepreneurs know plenty well that the best way to entice a sucker to voluntarily put his neck into a leash is to dangle a bit of bait, while disguising the leash as something other than a trap.
In Google’s case their bait is free geographic imagery and they have had the exquisite business sense to disguise their new browser by calling it “Google Earth” instead of what it really is, a browser for geographic content streamed out through public web sites.
If you disagree with what I write above, download the Manifold Toolbar and go visit Virtual Earth sites (you can also visit Google Earth sites, but you have to change the default URL used in the connection string if you are using default Google Earth image server modules). Voila! Internet Explorer browses Google Earth content just fine.
As we all know, Google hates Microsoft and is trying to figure out a way to sh0vel Microsoft out the door so they can replace that hated monopolist with a Google monopoly. Can’t blame them for trying, but you have to be pretty weak in the head not to realize that the reason they did Google Earth is to establish an alternative browser and to form the foundation for a web-within-the-web controlled by Google.
Earth was a good choice as geographic imagery is indeed very appealing, distinctive content that makes terrific bait.
ESRI apparently does not realize that its geobrowser is never going to get traction with anything other than torpid bureaucracies who cannot avoid falling into proprietary traps constructed by living fossil GIS companies. They obviously did it to convince folks like the Census Bureau not to take up with Google Earth. I suppose it has served its purpose there.
But if you really want to play roulette with the idea of sticking your head into someone else’s leach, it makes a lot more sense to join over 100 million other sporting types and go with Google. Or, if you want to make a deal that is business like where it is absolutely clear what everyone agrees to, go with Virtual Earth [which has straightforward, professional answers in every detail where Google does the double-talk thing].
And if you decide that the web is absolutely going to triumph over local applications for all sorts of content, including geographic data, well, then you should be working to use the existing framework of infinitely-supported APIs in the form of standard browsers like IE, Firefox and the like. There is absolutely nothing special about geographic data or analytics thereof that cannot be done within the context of a standard, non-proprietary browser if you really believe that is the way to go.
“I was just responding to I think misleading comments suggesting ESRI is sitting around not releasing anything new from version to version and SP to SP, because its just not true. Because of the breadth and depth of their line, add it up and they release lots of stuff each year spread across it all. They have been very busy latley, just like everybody else.”
I agree with you there, but I also respectfully submit that the above points, the ones you raised as examples, end up being very good examples of how ESRI is spending time badly. It si frittering away resources on creating a geobrowser to compete with Google Earth when it should be spending those resources on providing native 64-bit code in all ESRI products.
It is frittering away time on non-Windows work when it has yet to support Vista.
And, ultimately, it spends a vast percentage of money taken from customers on matters that are anything but engineering. That is why both in tempo, breadth and depth it is being clobbered by an upstart like Manifold.
I’m not saying it is easy for them to change. Once they have this legacy architecture, their legacy product line, their legacy business methodology and legacy economic structure hanging around their necks at every turn they find problems in adapting to modern times.
It reminds me of what was happening at Wang just before that company went out of business. They were clobbered on the one side by the eradication of minicomputers by PCs and they were clobbered on the other side by the looming eradication of their proprietary word processor business by desktop word processors like Word Perfect and Word. To the bitter end, Wang also was very busy doing all sorts of busy work spread across what was a very wide product line. But none of it dealt with the fundamental problem of competitors who provided ten times the capabilities for a fraction of the price. Sounds a lot like ESRI, doesn’t it?
Well I knew I was setting the ball on the tee there –
I do understand that Manifold thinks many of the things in the broad and deep ESRI stable are not worth while and so naturally not part of their business model, and they are doing good things, and I believe they probably have a wildly profitable business.
A clarification on this:
“- ability to do GIS in the “cloud” – meaning the Web API’s of ArcWeb Services (like google maps API, MS VE APIs, and the rest) – I am not aware of Manifold having anything similar?”
“Sure it does. There is the vast array of capabilities enabled through the Manifold API via IMS, which include things exactly like image servers (like GE), WMS servers, WFS-T servers and so on. “
… its not quite what I am getting at – I know you can set up your own Manifold IMS cloud, or pay someone to invent one for you, but there is no Manifold “cloud” provided by manifold – e.g. huge data center, redunant on the east and west coast or wherever, thousands of Manifold IMS servers, where I can connect to the GIS APIs across the Internet to create web applications for free, w/o installing Manifold IMS on my own wimpy server. So I can do this with ArcWeb Services w/o installing a lick of ESRI software on my computer, and paying $0 for the API to do it. ESRI even offers hosted managed services so I can even ride my web app itself on their servers (in addition to the server GIS already offloaded from my servers to theirs via ArcWeb services), as well as my data. It seems like a very modern approach, putting the entire GIS platform in the cloud, so that you don’t have to install anything on your own servers. I don’t think Manifold offers anything like that today, other than the means for someone to build their own implementation of the concept using manifold IMS – which I do believe some people are working on.
” huge data center, redunant on the east and west coast or wherever, thousands of Manifold IMS servers, where I can connect to the GIS APIs across the Internet to create web applications for free, w/o installing Manifold IMS on my own wimpy server. “
Free? Not at all true.
A very interesting scenario you pose above, but you don’t get the above with ESRI either. If what ESRI writes in their web pages about ArcWeb services is true, it is very much a for-pay service, and you get nickel and dimed in the usual ESRI way for a complex calculation of “credits” for just about every last little thing you or your visitors use, be it the data you use, the maps you create, the storage, the transactions, the hits, the reports printed, etc.
Speaking of “little things,” your base storage is only 50 MB. Pretty lame!
You don’t get “thousands” of servers either, you get to time share a much smaller farm that is no where near the multi-thousand server farms typical of something like Google or Virtual Earth.
[But so much is hearsay, as ESRI does not tell you in a straightforward way (or even in an oblique way) what the score is. I'd very much like to hear from someone who has actually decoded the ESRI pricing structure to post what the real costs are of Arc Web Services and what the real timeshared load is. ]
What we do know is that ESRI charges you $1250 for a block of credits, which is good for only one year, and that won’t last you long for any reasonably active web site.
Comparing the economics to a Manifold site, your $1250 with ESRI would get wiped out in a single day if you are doing volume like a Manifold site that gets 100,000 hits a day. Heck, it would be less than a day because that Manifold site could be running using many gigabytes of data (vast imagery is becoming typical) which won’t fit into the wimpy Arc Web Services ration of 50 MB of space, at least not unless you are willing to pay ESRI stratospheric fees. You could be paying more than $1250 a day while the Manifold guy’s costs are near zero.
But let’s take up the bigger issue. Is it really a good idea to host all GIS usage as web applications? Do people really want to do this?
They might if the alternative was to burn $50,000 on an ensemble of ESRI ArcIMS, ArcSDE, ArcINFO and the like, and then they’d have to pay someone another $50,000 to get it running for them, and then they’d have to pay ESRI an arm and a leg every year (thousands) for maintenance and then the ESRI solution ran so slow they’d need a double-handful of computers to host it. Yeah, sure, if you go down that legacy path just maybe you’d be willing to pay ArcWeb Services $1250 a day or a week to make the pain of administering ArcIMS go away.
But, suppose you are spending your own money? And, suppose you can get better performance for a fraction of the price, say $225 one time buy in, and use as much storage as you want without ESRI nickel and diming you at every turn? If you want someone else to administer the hardware, that’s easy to do with managed hosting for much less than the cost of the ArcWeb Service.
Arc Web Services is not the first “GIS web application hosted for you” offering, and like the others before it, it too has gone nowhere. It’s a good example of how ESRI chases after superficialities instead of improving the breadth and depth of its core product offerings. Why have ESRI’s predecessors failed at this and why is ESRI failing at it?
The answer is that GIS web applications tend to separate out into really simple location applications, like dealer locators or real estate listing applications, or they tend to be highly diverse and sophisticated applications like a town providing forward-facing web interfaces for a host of town data, or like the various applications for automating industrial scale vineyard management.
The competitive situation is that for simple applications people sign up with Google Earth or they use Virtual Earth. Those guys blow the doors off Arc Web Services for simple applications.
If someone is smart enough to think they can dodge the continuing costs of renting services from Google or Microsoft (for commercial sites), they are more than smart enough to run their own application without having to sign up to pay rent to ESRI.
People who have more sophisticated applications also tend not to use Arc Web Services, because if they are sophisticated enough to do truly sophisticated things they tend to be sophisticated enough to know how to outsource that part of it they don’t want to deal with (such as managed hosting) but they are also sophisticated enough to be able to handle (in fact, they *want* to handle that part of it) the technology they want to keep hands-on control over. No need there for Arc Web Services either.
It’s a myth that somewhere there is a city manager who is sophisticated enough to know exactly what GIS he or she wants to do, technically skilled enough to deal with remote ESRI API’s to put the application together at a distance but somehow so inexperienced he or she doesn’t know common sense moves such as how to outsource hardware to some managed hosting company. Doesn’t happen.
What does happen is that the guy who is challenged by any of that will be pretty clueless about putting together a sophisticated GIS web application as well regardless of where it is hosted. He or she is going to turn to a consultant to do it. And those consultants hands down choose Manifod, because that’s a way for them to make lots more money than if they rent overpriced stuff from ESRI.
They also like using Manifold because they can build applications of enormous power using all the tools they want (direct connections to SQL Server spatial anyone?), with vast data storage, endless programmability using the seemingly infinite resources of the Microsoft ecosystem and all sorts of other wonderful power moves without having to consult the ESRI credit calculator to see whether touching any of that will bankrupt them.
With Manifold you can put together a dozens of gigabytes capacity spatial DBMS system and serve 200,000 page views a day for $225 one-time cost for Manifold, zero cost for the DBMS and a web connection that costs you a couple hundred a month. And all that can be part of a dazzling web site that you can create and change as you like, integrate tightly with your organizational IT infrastructure and organizational data warehouses without paying anyone else a dime.
So the bottom line is that if you have a simple application and you believe in web applications, well, use the real deal and go with Google or Microsoft Virtual Earth. If you have a sophisticated GIS application you will have the capability and the desire to run it, outsourcing the tedious parts if that’s what you want to do.
If the ESRI response is that their Arc software is too expensive or difficult for even experts to use, and so you should be paying for Arc Web Services, well, I say that both options are no good. The right approach is to improve the software until it is truly full power and so low in cost it is effectively free. That changes everything, and makes for a really good set of options: if you want to do local GIS web applications that are highly sophisticated you can do them nearly for free. If you want to do simple applications via some third party web service, well, you can go straight to the real deal and use Virtual Earth or Google Earth.
That’s the financial vise that has clobbered all the “we’re going to host your GIS web apps so you don’t have to” business plans – no money in the simple applications and those capable of doing complex applications don’t pay rent to others for the privilege of being able to do the work they still have to do.
ESRI always has another gimmick up their sleeve. Rather than improve their base product, they come up with some hairbrained band-aid. Although in fairness to ESRI, they realize that the ecosystem they live in allows them to sell these band-aids to ignorant GIS users – so, Jack is making alot of money doing it that way (LIBRARIAN anyone? ARCSTORM anyone?).
“Free? Not at all true. “
- Sure, the ArcWeb Services APIs are free to get at and start programming with. Get a free 5000 credit account for messing around with. Call ESRI if you need more for testing with.
Also, check out ArcWeb Public Services which are a FREE subset of all services, that is quite functional.
So sure there are pay models, but there are also free models – so technically it is possible to stand up a GIS web application open to users on the Internet using ArcWeb Services w/o paying a dime to ESRI. I don’t think Manifold has anything similar? I also think ArcWeb Services APIs blow the VE APIs and google maps APIs out of the water – you just have to look at the APIs to see that. ArcWeb services is only missing VE’s 3D globe in the browser (Google Earth does not count because thats a geobrower, a separate app you have to install on a PC).
Now Dimitri – I am not trying to change your mind – and I know the futility of attempting a good spirited debate with you because that is not an activity one can generally win.
But am just trying to separate the reality from the spin – no matter how you word smith it, Manifold does not have a hosted distributed GIS service with APIs in the cloud that you access via the Internet. My ordering Manifold on a DVD and installing it on my web server to use IMS is not remotley similar to me purchasing a block of 100,000 commercial use / premium content credits for ArcWeb Services, or setting up a public service web site using ArcWeb Services for free. These are two totally different things. That’s all I am saying.
I understand that Manifold has not gone in this direction so far. I do see groups of guys buying Manifold IMS and setting up their own server farms, and selling space as distributed GIS in the “cloud” – but none so far that open up APIs to their services across the web, and none so far even remotley close to the robustness of AWS. Not to say that some upstart with some guys and some Manifold IMS servers can’t build the next great thing, maybe they will pull it off who knows.
Respectfully, I don’t get Tom’s last comment at all. It seems like pure silliness.
Regarding whether or not Arc Web Services are free or not… since we are talking about GIS applications done in a web cloud (where this sub-discussion came from) my assumption is we are discussing bona fide GIS applications done as a web application. That is definitely not free with ESRI.
[I respectfully disagree that publishing an API thta anyone can read or providing a limited trial is in any way "free" usage of the ultimate service, no more than when a car dealer lets you take a car for a test ride for "free" does that mean you will be able to buy a car at no charge.]
The details of ESRI’s offering are revealing. They have two types of service: Commercial Services and Public Services. Commercial services are the highly costly structure I referred to in my previous posting.
Their Public Services are not really GIS. If you read the description of what they are, you’ll see that a key limitation is that you *cannot* “Upload User-Defined Points, Lines, and Polygons” – well, doh!, that fairly definitively takes it out of the realm of what any of us real life GIS people consider GIS, don’t you think?
Look, it is up to ESRI to provide whatever service offerings they like and I don’t begrudge them that. But it is double-talk to refer to their Public Service as being a foundation for GIS applications on the web. Just ain’t so without the ability to “Upload User-Defined Points, Lines, and Polygons”. It may be free, but so are a lot of things that are not GIS.
For that matter, if you want “free” for public service applications, you can do so with Virtual Earth or Google Earth at zero charge. When it comes to serving billions of people worldwide, I think it is indisputable that both Google and Microsoft have a lot more credibility in the operation of server farms and in the provision of fast, reliable mashup hosts than ESRI.
Regarding the ESRI API vs. Google Earth or Virtual Earth: My point is not that either of the Earths have a bigger API than ESRI. My point is that they are perfect for most simple appolications, which is the bulk of interest for such web cloud users.
If you look at the millions of “mashups” [if you are really a believer in this web cloud thing you better get used to using their nomenclature...] currently done with Google you’ll see that by probably more than 10,000 to 1 people don’t give a hoot about ESRI APIs. They master the Google API and use that. If you believe in the worth of web clouds to present geospatial data you cannot ignore the reality that free Google mashups vastly out number any numbers using ESRI public service web offerings.
Consider it: there are tens of millions of mashups already done with Google. How many mashups have been done with ESRI’s Arc Web Public Services offering? A few hundred? This is a joke in comparison.
So the first part of my point was that ESRI’s bigger API is not a factor for most mashup users. The second part of my point is that those mashup users for whom a bigger API *is* a factor won’t choose ESRI either, because they don’t want to get on the hook for high charges when there are other options that give them more performance, greater storage capacity, more power and flexibility, greater control, etc.
“no matter how you word smith it, Manifold does not have a hosted distributed GIS service with APIs in the cloud that you access via the Internet. “
Well, I sure hope not!
I apologize if I gave the wrong impression – I had no intention of spinning it or wordsmithing it other wise. You are right, Manifold does not provide such a hosted distributed service. We don’t do that because we think it is a really bad idea that is a financial loser both for the company and for users. In fact, it is a double loser, because not only is it a bad idea from the straightforward tactical product side (customers have better choices than this), it is the sort of bad idea that sucks resources out of a company and prevents them from being invested into products that provide a really worthwhile benefit.
I think reasonable people can disagree about this, but so far it is clear by overwhelming margins (one might argue whether it is a factor of 1000 or a factor of 10,000 or more, but it is overwhelming), truly epic margins, that ESRI has lost to both Google and Microsoft in the mashup arena. In fact, ESRI is so totally miniscule compared to Google or Microsoft mashup usage that it is somewhat of an inaccurate use of English to even suggest that ESRI “competes” with them. It is a bit like suggesting that a pick-up soccer game among schoolchildren competes with World Cup finalists.
Google Earth and Microsoft Virtual Earth are totally dominant in the case of almost all “GIS” applications being done in web clouds. That’s so because, as I wrote in my earlier posting, almost all such applications are relatively simple ones.
So, yeah, sure, I’d be the first to say that Manifold has had the common sense to realize it is really stupid to go off and try to blow Google Earth or Virtual Earth away in that 99.999% of “geospatial web cloud” applications market.
Unlike ESRI, instead of trying to offer an alternative to the “Earths” we have embraced them. Instead, Manifold automatically integrates with such image servers, allowing people to leverage the power of such web clouds within Manifold applications. See, for example, the very nice visuals in the tutorial at
http://www.manifold.net/doc/exporting_kml_to_google_earth.htm
It’s no accident that Manifold has become the number one tool for creating KML/KMZ for all those zillions of people beavering away at creating Google mashups. By investing into real GIS and embracing web based services (like image servers and geocoding servers) as an integral part of Manifold we have taken what is useful from such services for the use of real GIS users.
“I do see groups of guys buying Manifold IMS and setting up their own server farms, and selling space as distributed GIS in the “cloud” – but none so far that open up APIs to their services across the web, and none so far even remotley close to the robustness of AWS. “
First, I don’t think anyone has statistics either on the robustness of AWS or on the robustness of server farms set up by various entreprenuers so it is not really possible to say with accuracy which is most robust. What we do know is that ESRI software is famously not robust and famously not 64-bit, while Manifold software run in 64-bits is famously bulletproof, the most robust GIS stuff on the planet. Unless ESRI is running AWS on 64-bit Manifold product, it is almost certain that, given an equivalently competent management of server hardware, the Manifold solution will continue to be famously bulletproof and the ESRI solution will continue to be characteristically crash-prone. I’m not telling any experienced ESRI user something they don’t already know from personal experience.
The second point I would respectfully make is that Manifold consultants do not typically set out to go into the generic web cloud business because they too see it as a financial loser both for the vendor and the customer.
There are plenty of Manifold consultants doing web sites for people, but they are doing it using the rather well established path for such things. They are simply using Manifold instead of ArcIMS or whatever to reduce their costs, be more competitive, achieve higher reliability and get more performance.
And when you think about it in close detail, for the sort of expert users who create bona fide GIS web applications, the whole “web cloud” thing is a red herring.
Really, other than your rental costs, what is the difference between a) hiring a managed hosting company and b) hiring ESRI AWS? In both cases, you never even see the server that runs your software. You sit at a console and craft a web application using a vast API and the thing comes up and runs and someone else manages the server farm.
The difference is that for managed hosting you are paying under $500 a month for huge bandwidth and capacity, and you have just one extra step: uploading your $225 Manifold Universal Runtime license into your managed host. Takes five minutes and if you know how to use a sophisticated API to create a GIS web application you certainly know how to do that.
There may be some folks out there who think that ESRI knows how to run managed hosting better than the very large, very experienced managed hosting companies. I respectfully disagree. I’d bet a fair sum that ESRI doesn’t even run their own servers but subcontracts that out to a real managed hosting company. So why pay a middleman when you can deal direct with that managed hosting company and get the benefits of a direct relationship?
Last, but not least, there are many possibilities with direct managed hosting relationships you won’t get with ESRI. For example, if you want you can load up your host with an NVIDIA CUDA card (or cards). Run 256 processors to perform computations absolutely instantaneously. I grant that right now the number of web-facing applications that can use such computational power are few, but as Manifold expands the use of CUDA the ability to compute and render instantaneously will expand the performance lead of Manifold IMS even further. Can’t do any of that with AWS.
In response to Tom’s last comment, it does appear that ESRI comes up with stuff out of left field at times. But the reality is that whatever is added or modified in the product offerings is largely done in response to customer needs and feedback (lots of screaming, nashing of teeth, etc!). This includes new functionality as well as enhancements and critical bug fixes.
May be hard to believe, but its the truth. There is a lot of good intention, but the agility that’s required to respond in an effective and timely manner is just not always possible there.
you hit it right on!