I don’t like the term “ESRI Shop”. Both open source developers and ESRI developers use it as an excuse one way or another. Just because you are running ArcMap or ArcGIS Server, doesn’t mean that you can’t explore open source GIS software and in fact you might be surprised how it makes your ESRI ArcGIS software even better.
A couple months ago, one of my clients asked me how they could bring KML into their ArcGIS Desktop maps. We’ve been through this before so I won’t detail my feelings on the subject, but ArcMap cannot read KML without some help. We looked at a couple different solutions and FeatureServer seemed to be a perfect fit for what they wanted. They’ve got people digitizing points/lines/polygons in Google Earth pro and getting that data into their SDE databases needed to be easier. Then another client wanted to digitize polygons inside ArcGIS Server, but they couldn’t afford a license for ArcGIS Server. Again FeatureServer seemed to be a great choice, coupled with OpenLayers. The idea seemed to come together at that point. Use OpenLayers to allow people to create and use FeatureServer as the translator to move the data between KML, GML, PostGIS and eventually ArcSDE. The idea is to let folks use the tools they are most comfortable with, whether it be ArcMap, Google Earth, or any OGC complaint application.
Now the concept sounded perfect but we ran into some problems. First off ArcMap still couldn’t read KML without help (and I could not be assured that an extension would be installed on ever workstation) so GML sounded like the perfect solution. ArcGIS Desktop can read “GML Simple Features”, but what FeatureServer was giving out was more complicated. You could read the GML from FeatureServer with Data Interop extension, but without it, ArcMap gave an error. Thanks to Christopher Schmidt, we were able figure out the schema for the ArcGIS GML “format” and any data that was digitized in OpenLayers could be downloaded as GML and viewed in ArcMap. That was huge because now you can view any features in OpenLayers, Google Earth and ArcMap which is exactly what they wanted.
What is really important to understand here is this was all done with open source software (FeatureServer, OpenLayers, PostGIS) and was developed quicker than an ArcGIS Server application could have been done. Christopher was obviously a huge help and the prototype couldn’t have been developed without him figuring out how to get FeatureServer to interop with ArcGIS. There is no overhead involved and no licensing fees/maintenance that needs to be paid.
Having a group of GIS techs digitizing PDF maps that people have marked up is not cost effective for anyone. Having people remotely digitize features and be able to work with them in PostGIS, Google Earth, ArcGIS and just about any OGC complaint application is very powerful. It really comes down to using open source to make everyones job easier and doing much more than you could before with much less money. When you take out the maintenance and the licensing of server software, you can focus your money directly on the problem and solution.
The next step is to get FeatureServer to write directly to ArcSDE (or as we now call it ArcGIS Server Enterprise). That way there is no importing of GML (or any other data layers) into your ESRI workflows. FeatureServer takes care of everything and you just interact with you spatial database like you always have. I’ve heard that OGR support for ArcSDE write is on its way so that will open FeatureServer up to many more folks.
That brings up a huge issues for many folks. How do I get my ArcSDE datasets out to the public so they can use them? Imagine FeatureServer just sitting there offering up KML, GML, WFS, GeoJSON, OpenStreetMap .OSM for all to use. I love this approach to getting data in and out of ArcSDE; quick, easy, cheap and it just works.
BRILLIANT!
83 Comments
What a great solution James. I like how the workflow is so simple. This is what ESRI should have done years ago.
Neat stuff. I like how you can use the software you want to use and share with everyone. I agree, why isn’t this already available in ArcGIS?
I can’t believe you are calling the ESRI GIS Portal Toolkit slow, expensive and hard to use.
You can chalk up our organization as one that just rolled it out a couple weeks ago and now realizes that its crap (well the rank and file know this, the leaders who have dinner with Jack seem to not care). As a taxpayer you should be outraged!
I like the approach here. Simple solutions to simple problems. I think we loose sight of that in the GIS world.
Very interesting stuff. I have just started following your blog, which I’m loving BTW. Question regarding this topic. It seems that the proposed type of editing is rather simple (not meant in a derogatory way), kind of like editing in ArcView 3.x. Is there any support for snapping?
It appears in your example images the client needs to digitize building footprints from an aerial photo and may not need snapping capabilities. However for proper data creation it seems snapping would be an important feature.
Yes you can perform snapping. You can check out the demo here for an example:
http://dev.openlayers.org/sandbox/camptocamp/feature/examples/modify-feature.html
You can edit the polygon by clicking on it and moving the nodes or draw a new polygon and have it snap to those same nodes.
We choose not to implement this right now because it just isn’t needed. Though as more polygons get created, being able to modify them and snap them will become very important.
You can even take it one step further and perform basic GIS analysis:
http://crschmidt.net/mapping/wpserverdemo/
What is nice is we can start with a very simple application and add functionality as needed rather than start with the kitchen sink and strip things away.
@James, “What is nice is we can start with a very simple application and add functionality as needed rather than start with the kitchen sink and strip things away”
I couldn’t have said it better myself. My organization has a need for a simple online polygon editing system. This type of solution would make a lot of sense, but it takes imagination and commitment, and it’s easier for many people to believe that AGS Advanced, out of the box, will cure all our problems.
I’d be anxious to try a similar setup using Spatial SQL against the SDE (if I may still call it that), which is supposed to support versioned updates if you structure your SQL properly. If only I could get that extension installed.
Very nice. I like the flow.
Of course, we often run into the “ESRI shop” (or “Autodesk shop” for that matter) syndrome in the fact that the IT and/or GIS dept. won’t let us stray from what they already have, which is quite short sighted if you ask me:-(
There is a COTs web solution available that offers data editing in SDE via an ArcIMS front end.
http://demos.geocortex.net/imf-5.1.006-es/sites/demo_editing_suite/jsp/launch.jsp
There is a COTs web solution available that offers data editing in SDE via an ArcIMS front end.
http://demos.geocortex.net/imf-5.1.006-es/sites/demo_editing_suite/jsp/launch.jsp
It’s not free but there is something to be said for leveraging the distribution of development and maintenance cost and risk over a large client base. I certainly couldn’t justify paying for the staff or vendor support to build and support an application of this nature.
argh… sorry for the redundant text. Not sure what happened there.
James, you’re beginning to preach. Don’t believe, that your solution leads to the thesis that’s it is developed quicker, more focussed and all in all cheaper, give us well structured arguments.
1.) it is cheap because you don’t have to pay license fees and Christopher has helped you out without any costs. What saves costs behind that? What does it depend on? Friendship, community?
2.) is it quick because of the same reasons? is there a dependency to the knowledge level of support? are small software clusters (like christopher and you) more agile and better prepared than big ones (like esri support)? where are the depend parameters?
3.) is it a good technical concept because it’s based on more or less official standards? does standards have something to do with the marketing concept of the software vendors? where are the “standards”? where are they missing?
Please, again: Don’t put buzzwords together. Keep the discussion clear. Several reasons are mixed up in your argumentation. First of all it was cheap, second of all you’ve had a agile, flexible community, third of all, you’ve had a underlying protocol.
I’m missing clarity here at all sides.
Remotely editing of data is now one of the keys to providing efficient workflows, and ESRI technology is not always the first choice. Open Source, Free (of cost) and Free is one choice, but as some correctly notice there are other solutions out there. Our company has developed a remote editing solution for Oracle, WFS-T, ArcSDE and PostGIS, and the OGC standard will help others like us to put solutions on the market.
You will find versions ranging from free to 3000€/year on http://www.spatialedit.com.
ESRI and others price policy paves the roads for both Open Source and companies like Manifold and us to address certain market segments.
Looking at the images, I just realised why ArcGIS cost more than Google Earth – They put more car into the images
@bender: You’ll just have to wait for tomorrows post on that.
@timmy,
James did use off-the-shelf components, there just wasn’t a capital cost associated with purchasing them. IMF is a off-the-shelf component, so are Openlayers and Featureserver. Why is the integration cost of the components James chose so much more galling than the integration cost of bringing IMF into a shop?
On your second point, what makes you think Openlayers and Featureserver aren’t leveraging development over a large community?
@timmy
Anyone putting money down on an ArcIMS solution in 2008 might as well toss it in the trash.
My clients realize that ArcGIS server is the modern way to go. In this case they did not want to spend the money on services they didn’t need. This could have been done with AGS, but we accomplished the same thing without spending any extra money on licencing. Plus OL & FS were improved.
Beers all around.
@ paul
I’ll admit the concept of open source has my attention technically and is an intriguing market model but there are some strategic points that I can’t seem to reconcile, perhaps you all can clue me in…
These are sincere, and perhaps naïve, questions; I’m not trying to prove a point and pick a fight rather I’m simply trying to develop an intelligent perspective.
@ James
You have a point at first blush… and you’re right there is no “one size fits all” application out there. Which makes me surprised to read your blanket comment about IMS. Many clients need solutions now and the business requirement that the solution is designed to support will lose much much more if it isn’t in hand yesterday. So… do they eat the loss as they wait… and wait… and wait… for something else or buy something proven that is immediately available that can be deployed without additional layers on their enterprise stack?
So far I haven’t seen any application as stable, robust, and efficient when it comes to editing online. If someone knows of an application or can build an application with its capability AND support for less that what they’re selling it for…. I have cash in hand and am all ears.
Seriously.
@timmy: My comment stems from the fact that multiple ESRI devs have told me that for all intents and purposes, ArcIMS is in maintenance mode. It doesn’t support any new features and probably won’t. Investing in such a product means that you won’t be able to invest in future more modern products because you’ll be tied to the past. ArcView 3.x works, but you wouldn’t drop thousands on dollars on a solution (or at least I how you wouldn’t) and the same goes for ArcIMS.
Editing vectors in OpenLayers is as “feature rich” as anything I’ve ever seen on the ESRI side.
@ james
You raise a good point and you’re right.
If I developed the application myself or had someone develop the application for me I would be out of luck when ESRI drops IMS support (which I would argue will be quite some time if 3.x is any indication). But that’s my point… I want to buy the COTs application with a large customer base thus leveraging economies of scale to reduce risk and cost and …if I’m savvy enough… make sure the maintenance covers upgrades that will accommodate the inevitable evolution of the backend technology. If I’m even smarter… I would pay the same vendor for any customization that is required and work with them to include it in the next release or sell it as an extension insuring that my maintenance covers the cost and reduces the risk of the inevitable software upgrade.
As for OpenLayers are there any “plug and play” products out there or would they have to be built from scratch?
OpenLayers is a Javascript API in the same sense that Google Map and Virtual Earth are. What types of products are you talking about here? FeatureServer, TilesCache, ArcGIS Server, ArcIMS, etc… all “plug and play” server products.
Or am I not understanding you.
Maybe if I put it this way. I don’t want to develop the software that meets my requirements, I want to buy it as a complete product. I understand OpenLayers is an API that can pull in data from multiple sources.
In this post you noted that your client wants you to build something to meet their business requirements and you’re using openlayers to develop it. It’s the end product that you would provide to them that I’m talking about. However I want to avoid sinking money into building what amounts to an exclusive application with high exposure to risk to get what I need. I want to buy it and mitigate risk by paying for maintenance. A good example would be the third party client applications that sit on top of ArcIMS (i.e. GeoCortex, Freeance, GeoSmart, et). It is very improbable that someone could exclusively build any one of those applications for what you would pay for them off the shelf. Why? Because the cost is spread out over the entire customer base. Its COTs applications such as these that leverage the open source resources that you’ve discussed that I’m interested in. Make sense?
Good Article
I haven’t yet understood what people want from ‘plug and play’ editing, but a feature list would at least let me understand whether OpenLayers can do it, and maybe even start building it (if there is an it to build).
Just FYI, GeoServer 1.6.x has ArcSDE write capabilities, and OpenLayers talks directly to it with WFS-T (and soon we’re hoping to implement REST like FeatureServer does).
We’ve also got some versioning of features with a PostGIS backend. You can check out the demo.
The UI is a bit rough, but we’re working on improving it. With some additional funding we’d love to hook it up to ArcSDE’s versioning. We also have another beta demo with a slicker UI, but less focused on ‘GIS’ needs is at
@timmy: We’re working a lot on online editing, contributing the capabilities to OpenLayers and GeoServer. I don’t know how much Arc* web editing costs, but we build applications and sell commercial support on GeoServer and OpenLayers. Though we’re full on work right now so may be a couple months till we could get started.
@timmy: There isn’t anything custom there at all other than the features in PostGIS and the HTML eye candy around the map.
There was some coding that Christopher had to do to get the GML to work with ArcGIS, but now that is done any project (not just mine) can take advantage of that. None of it is custom.
@Chris Holmes Yea we’ve started looking at GeoServer as well. I think GeoServer might be more comfortable for ESRI Server users than MapServer because of its configuration. Much closer to what ESRI users are used to using over MapServer.
@James Fee: “Anyone putting money down on an ArcIMS solution in 2008 might as well toss it in the trash.”
I’m sticking with ArcIMS AXL services unless something like feature linked anno or complex custom symbols are needed. AXL image service are clean and simple. AGS is single threaded and a total performance dog.
Without TileCache ArcIMS is slower than ArcGIS Server with its MapCache.
Having to request a new image every time you change the map is very slow.
@ James Fee: It seems as if you are missing timmy’s point all together…or maybe we are missing yours. I am 100% in timmy’s position. I need a product I can buy and easily implement and migrate. I can’t afford to be constantly tweaking things and getting under the hood of an application every time I need to add a connection to a new database, create a new search or report, or add some new functionality. I need proven supported software. Our resources are too scares to spend this kind of time playing around with things.
Some organizations can quickly deploy the types of application you develop. These would likely be small more agile organizations. Larger less agile organizations (government, utilities, etc.) need proven more established applications with maintenance plans and migration paths.
Now don’t get me wrong. I’m glad…very glad…there are people like you out there working on stuff like this. Because soon these ideas are going to make their way into products that work for my organization. The hope is these products will be better, faster, cheaper.
Some here have poo pooed the idea of open source. I see it as a development process by the public rather than a software company. In the end we as users and software consumers will all benefit. So keep up the good work!
@James Fee: “Without TileCache ArcIMS is slower than ArcGIS Server with its MapCache.
Having to request a new image every time you change the map is very slow.”
We have developed tile caching for ArcIMS image services. It’s kind of beta as a pet project for our developer while he takes the bus home. It is really dang fast.
By Timmy’s logic, your tile cacheing isn’t “off the shelf” and not deployable in his world.
You can actually use TileCache.org to handle the caches with ArcIMS.
With your comment above, what you describe seems to be filled by open source software. Very agile stuff.
“By Timmy’s logic, your tile cacheing isn’t “off the shelf” and not deployable in his world.”
No I totally agree. I have no idea how or if we can deploy it, which sucks because it’s pretty cool.
OK, I think we are arguing the same points. Buzzwords get in the way of innovation which is probably why the interesting stuff is in open source and open APIs such as Google Maps.
We need to step away from these words in the ESRI world and focus on products.
“We need to step away from these words in the ESRI world and focus on products.”
Well said, James, although it would have been even better said if you enumerated those products which will do what you want in this thread for absurdly low cost, no self-development required.
I have to admit, though, that I feel sympathy for you in the original blog posting for this thread – it is like you have been so abused by the limitations of the ESRI software you use that you are willing to do backflips through hoops, and feel happy about it no less, to accomplish what should be a natural, built-in part of the software you use.
Doing the GIS equivalent of cobbling up your own shoes in this case is fine if your hobby is software development and integration and maintenance (the real cost over time) and you don’t care how much time that swallows, but it is totally off the charts in the common sense department if you have a business to run. Anyone who is happy about spending so much money on ArcGIS thingies yet still having to do stratospherically expensive self-development like this probably is not spending his or her own money.
Manifold solves everything you describe in this thread in the off-the-shelf Enterprise edition for $395. Two visual examples are even online:
http://www.manifold.net/doc/example_tracing_virtual_earth_into_sql_server_2008.htm
This shows using Virtual Earth to create vector layers directly into SQL Server 2008 spatial (Katmai, the new Microsoft product ESRI still does not support), but it could have just as easily been Google Earth and SDE (both of which can be connected to in an identical fashion by Manifold).
http://www.manifold.net/doc/exporting_kml_to_google_earth.htm
This shows the use of Virtual Earth to create data for a KML file. Manifold reads and writes KML, of course.
If you want to have many thousands of people subsequently editing against a centralized data store, of course you can do that as well.
I should point out that besides the initial lower cost of buying well-debugged functionality off the shelf you also get the continued benefit of someone else maintaining that functionality so that everything keeps working well together even as technology changes (new Windows releases, etc., not that ESRI cares about those… Vista anyone?)
good points
@Dimitri:
My choice of open source tools has more to due with the requirements set forth by the client, not any need to avoid ESRI. I’m able to grab tools that I need to meet those requirements and still integrate them in with workflows that are ESRI centric.
Something very powerful there. ESRI or not is not a point to take away from this project. It is meeting their requirements exactly by using tools that give us the freedom to do what we need with zero overhead.
Let’s have a long posting for a change…
About the jargon: “COTS” means “Commercial Off The Shelf,” as in a commercial package sold in standard production, that is, not a custom package developed as a “one off” and not freeware.
James:
Let me translate that into words less fraught with political/religious overtones: “I could not do what my client required me to do with the ESRI tools I had. So I used some other tools in addition to the ESRI tools I had.” That’s the essence of the matter, don’t you agree?
I understand that you are not saying that you made some political decision to “avoid ESRI”. It is clear that your words simply mean, no more and no less, than the ESRI tools you had did not meet the requirements of your client. So you went out and got some other tools that did. That is your point and I can see that.
But, why would you reach for “open source” tools with such evident commitment when you could have done the same thing, perhaps even much better, for an utterly trivial cost using COTS software? Could it be that while you will not allow yourself to fall into the mistake of “avoiding ESRI” just for the sake of avoiding ESRI, nonetheless you will eagerly dive into the equivalent mistake of “avoiding COTS” in the case of outstanding, high performance COTS software that happens not to be ESRI?
You are conveniently ignoring my point, which is that there is nothing about your situation that compelled you to use open source tools instead of, say, COTS tools that are available for free but which do not provide source code access, COTS tools that are expensive but still cheaper than the labor time you expended, COTS tools that are inexpensive and dramatically less cost than the labor time you expended and so forth. Quite obviously, COTS tools available in GIS will enable you to maintain your ESRI-centric workflow. Manifold, for example, is happy to play with SDE data sets.
Nothing at all about your situation compelled you to use open source tools instead of commercial tools that could have done the same job much better with much lower configuration time required. You can quibble about the costs involved, but I think experienced people realize perfectly well that a commercial tool which costs almost nothing but which saves many hours, days or weeks of time and which does a job much “better” (in the sense, perhaps, that “better” is commonly used in GIS, for example, to mean that it is more performant, more reliable and much easier to maintain into the future) is hands down the wiser choice than, say, an open source toolkit which may be “free” but which costs far more time to utilize and is not “better” at the task.
If what I write is true, that a package selling for a few hundred dollars could have accomplished all this out of the box, for most people it would have been incredibly unwise to have a) wasted all that money on an ESRI platform that failed in the first place and then b) ignored the opportunity to grab a COTS solution which at an astonishingly low price would have saved them from launching their own mini software development / integration engineering effort.
Many people would wonder why you would cling to “workflows that are ESRI centric” if dong so requires you to either keep buying other people’s commercial GIS products or to launch your own software development / integration projects using open source modules in an attempt to fill in the holes of what ESRI is not able to do. Even if the COTS products you use are low in cost, and even if you discount your costs with open source, in either case the question naturally arises, to quote your words in a different context, why you would cling to a vendor that does not allow you to meet the needs of your projects directly but instead imposes workarounds upon you?
I take the above at face value, but you are asking us to make some logical back flips here unless you know something about “back room” requirements here that you’ve not told us.
Quite obviously, if your massive investment in ESRI software failed to meet the requirements of the project then it very much is a question of “ESRI or not,” at least as far as this particular project evidences. What other lesson can you take away from this project in the “ESRI or not” department? That your choice of ESRI was correct because it did everything you needed without workarounds? That it didn’t matter that you had chosen ESRI?
If you could have used a different vendor’s software from the beginning so that your client’s project needs could be served directly without any workarounds, you bet it is a classic example that your initial choice of ESRI was wrong, dead wrong, especially if using some other vendor’s software from the beginning could have saved you lots of money over using ESRI.
For that matter, suppose you could have done the entire thing from the beginning with open source instead of using ESRI software. In that case also, it would be clear that your choice of ESRI from the beginning was the wrong choice.
I can understand that if for political reasons, perhaps a pre-existing political commitment to using ESRI you had no choice but to make it look as if the project was being done on ESRI software. In real life, that happens all the time so I agree maybe that is what is going on.
If that is what is going on in this case, well, then it makes perfect sense to me that you would not want folks to look closely at why on Earth were you using ESRI stuff in the first place and thus got yourself caught in the jam of having to hack up workarounds using open source components, to fill in with such workarounds what ESRI could not do but which any sensible, modern GIS would do from the get go.
But if that is what is going on, that has nothing to do with a choice between commercial and open source software. It is just a common, grubby case of politics forcing you to make it look like you are working with ESRI stuff when in fact you need to use some other tools to get the job done.
For that matter, if it is politics in play could that be the real reason you could not choose a more effective COTS package instead of having to jump through hoops with an open source develoment project? Could it be what is really going on is the politics of the matter do not allow you to use a much better COTS package because doing that might make it clear that the choice of ESRI was a mistake in the first place? If that’s what it is, well, I understand that is how it goes. But let’s not use your suffering because of that as a way of somehow suggesting that open source tools in this particular case were as a matter of technical or business efficiency the best choice.
James, the point of using whatever toolset you choose is to choose tools that give you the freedom to exactly meet requirements, hopefully with zero overhead. Nothing about that happy ideal points one way or the other between commercial tools or open source tools. Your statement is simply the rather banal one that you should choose the right tools for the job.
In this case, your initial choice of ESRI tools was not the right tools to give you the freedom to meet the requirements of the job, so you supplemented that by choosing additonal tools, in this case “open source” tools.
Your statement does, however, for all of its obvious banality does include one massive bear trap just waiting to clamp down upon the unwary: this interesting notion of “zero overhead” you introduce.
I trust that given your stylish and effective writing style you didn’t mean that literally. I know from meeting you personally that you are an intelligent and insightful guy who is not the sort to be easily fooled the way inexperienced beginners so often are. So, for example, even though you and I have never discussed your background in physics, there is no doubt in my mind that your general experience and intelligence would save you from believing in, say, perpetual motion machines.
So that’s why there is no doubt in my mind that you know full well that there is no such thing as a perpetual motion machine in software, either, and thus you did not mean the words “zero overhead” literally. I know you know perfectly well there is always overhead of some kind and that it is just a matter of choosing what kind of overhead one prefers.
So my inference is that by “zero overhead” you meant one of two things. You either meant that to say “OK, I knew I had additional costs but they were bearable.” Or, perhaps you mean that phrase to mean, “I can bury these costs where they will not be seen, because only I know the labor I spent on this.”
One last thing: nothing about your reported development process in any way takes advantage of the “open source” nature of the tools you used. As far as I can see, you did not do any source code reprogramming of the tools you used. You simply employed them as “black box” modules using their APIs as one normally would utilize any objects or modules provided by someone else. Because in no way did you touch the source code for the tools you used, the “open source” aspect of it did not come into play at all. You could have just as easily been using commercal tools.
The above points out a common point of conceptual confusion within many “open source” tools discussions. The talk often mushes together in a confused, inarticulate way several different characteristics of software that are in fact quite separate entities:
Is source code available or not for the software? – There are plenty of commercial packages that make source code available, for maintenence, for education, for customization, to reduce risk through escrow, etc. , and they do so over a very wide range of terms and conditions, from darned near close to public domain to tightly controlled.
Are you allowed to use this package at all, either in source code or in binary? – Both commercial and open source packages will often place significant limitations on how you can use either the source code or binaries.
Source code access is not just something unique to “open source” software, and, as the GNU license makes clear, it is not just commercial software that makes source code available with terms and conditions attached. People can quibble all they like about whether the terms and conditions in a particular case are benign or malicious, tolerable or intolerable, nobel in purpose or greedily base, etc., but it is indisuptable that in almost all cases source code provided either for “open source” or for “commercial” products comes with terms and conditions attached. Likewise, in almost all cases there are significant limitations on how even binaries can be used. Many “open” packages, for example, do not allow commercial use.
[For that matter, if you are working in a commercial setting (you said the word "client" so I assume you might be), you also have the labor/legal cost of parsing the various licensing agreements of the different bits and pieces you have chosen to integrate to see if you can even use those "open source" bits in your commercial application. That is sometimes not so easy, as I invite anyone who is not familiar with OpenLayers to see for themselves: go to the http://www.openlayers.org/ site and see for yourself just how long it takes you to even find a license let alone how long it takes you to figure out exactly the limits of commercial activity you can do with it. This is not easy for someone completely new to that module. I realize that giving the faintest flying hoot about licensing is not something that script kiddies care about, but it is something that responsble folks like James and you, the gentle reader, and I care about.]
Is the product free of cost? – Plenty of commercial packages are completely free of any licensing cost whatsoever. Visual Studio Express, SQL Server Express, DB2 Express with Spatial Extender and similar are completely free of charge. Conversely, plenty of “open source” packages cost money to acquire, packaged Red Hat Linux to name just one.
Can the package be freely redistributed? Surprisingly many open source packages cannot be freely redistributed. For example, if you are a “commercial” or “for profit” user in many cases you cannot redistribute or even utilize some of the open source packages, while quite many commercial packages are far more freely redistributable. A “commercial” package, SQL Server Express, for example, is actually free of cost and freer to redistribute than some versions of the “open source” package, MySQL.
Can you make source code modifications without losing rights? – It is no accident that many “open source” products are copyrighted so that if you do indeed exercise your contracted rights (the GNU license so often used is indeed a contract) to make modifications you can lose other contracted rights you have paid for, typically support rights or the rights to even refer to the modified product by its trademarked name. It is no accident that many successful open source products are developed and controlled by consortia which do not allow just anyone to willy-nilly say that what they have created is that product using the trademarked name.
All the above play into many cases where people say the words “open source” without having a clue as to what they are talking about, or whether what they mean is “I could download it for free from the web” and not at all access to source code or, even, the ability to lawfully use it for a particular client. For example, surprisingly many “open source” products are not licensed for use in for-profit enterprises, even if the “for profit” is a nice guy like James doing a bit of GIS consulting for a local small business.
James, in your case you seem to have confused “free of initial cost” with “my access to source code made this possible.” You used the capabilities of the tools exactly as they were. You didn’t invent new tools or extend the existing capabilities of the tools you used, you simply used them as they were. The theoretical availability of re-engineering the innards didn’t play into what you did at all.
The only thing that mattered was a) what the capabilities were that you needed to fill the holes in your initially wrong choice of ESRI and b) how much would it cost you to fill those holes using the tools you chose.
For that latter point, I trust no one reading this blog is so incredibly inexperienced as to fail to realize that the acquisition costs of software are usually by far the lowest cost involved in using it. Even in the case of excessively high legacy pricing like ESRI’s, the cost of the software is usually dwarfed by the costs of configuring, integrating, deploying and maintaining the application.
I wish, therefore, James that in your excitement about your use of “open source tools” in this particular example you would acknowledge that open source code access played no role whatsoever in your utilization and that you did indeed have nonzero overhead, namely the very high labor cost of configuration, integration, application development, deployment and, forever into the future while the application lives, maintenance. [We'll set aside legal costs since many open source advocates routinely ignore them anyway.]
If the end result works for you, fine. But let’s not have any of this pretense that choosing the wrong tools in the first place followed by a labor-intensive workaround process that ignores all costs except initial licensing is remotely as cost effective as simply choosing the right COTS solution, at dramatically lower cost, which would have given you the freedom to address your project’s needs directly without any workarounds.
I am no more against the use of open source tools than I am against the use of commercial tools. But I do hope that in either case folks consider all factors involved and that in either the case of “open source” tools or “commercial” tools there can be traps for the unwary which should be considered.
I would be interested in hearing from James exactly what it was his use of this particular set of open tools allowed him to do that he could not have done with much greater efficiency using a modern, low cost commercial GIS package. He did not use source code access, so we know it was nothing to do with that. What could it have been?
Could anyone actually read all that. I could only read about 1/5th of it.
James, lets put this to bed right now. Please list the requirements of your application and then see if Dimitri can recreate it in Manifold. If he can, I’ll buy Manifold – if he can’t, then you should ban him from this blog.
Actually, jonny, I have already pointed out waaay back in the beginning of the thread that all this could indeed be accomplished with Manifold. Thanks also for your generosity of spirit in that people who disagree with you should be banned from conversation. That really shows you have a lot of faith in your views.
My posting sought to explore meaning of James’ comments, mainly “It is meeting their requirements exactly by using tools that give us the freedom to do what we need with zero overhead,” and how that statement does or does not explain why it makes sense to use open source tools instead of commercial tools.
I’d especially like to understand why James presented his use of open source tools in a way that seems to indicate he did not think that commercial tools could have been used to solve his problem. I suspect that James simply was very pessimistic because his ESRI tools could not do the job, so I wrote to encourage him not to think that just because the ESRI tools were not up to the task to assume that his only way out was to ignore all other commercial tools.
I am deeply curious what acknowledged experts like James find useful in “open source GIS.” In this case, is it because James is reacting to the limitations of an ESRI toolset or is it because open source GIS tools have some meaning beyond a reaction to the limits of legacy products?
But even with experienced, serious people of indisputable good faith like James it has been difficult to nail down exactly why they use open source, because (as can be seen in these recent threads) you get circular reasoning in response to key questions.
Someone says “I use open source because it is free, it has zero overhead.” I say, “well, it’s not hardly free there are costs involved, plus you know there are plenty of commercial packages that are free and have less overhead.”
So they say “Oh, no, I don’t mean “free” as in free of cost I mean “free” as in “freedom” to use my source code access to adapt this tool to whatever I need it to do, so that I can meet the needs of my project directly, without any workarounds.”
And then when I point out, “You haven’t used source code to adapt the innards of what you are using, but instead you are using it just as you would commercial code… anyway, if you had adapted the the source code isn’t that a greater ‘workaround’ than the open doors a quality commercial product would give you?”
… and that’s when the justification gets back to “But, it is free of cost, you know, licensed through GNU” …and then we go through the whole thing again but with a bit more said about licensing.
If people play this game of “Oh, it’s that it is free of cost” oscillating with “Oh, no, it’s that it is free to modify the source” they never actually give you a cogent reason why they are using it even though they are implying at both turns a gross misconception about commercial software: either that commercial software has to cost you more or that commercial software can’t ever allow you to utilize whatever modifications you see fit.
James’ comment:
“It is meeting their requirements exactly by using tools that give us the freedom to do what we need with zero overhead.”
…in his usual succinct style has distilled down both such justifications into a single, concise sentence.
The declaritive purity of that sentence in expressing the case for open source GIS makes it a highly useful starting point to see if the intended meaning holds for most people doing GIS on a day-in, day-out basis. I say that for most people for most tasks on most days neither the ability to modify source code nor the implication that there is zero overhead is true. Frankly, I don’t think it is even true on this particular day for James and his task.
It could be that James’ use of open source tools for his project was nothing more than the curiosity of a gifted mind that found this interesting, that had a friend who was willing to provide expert technical services free, that this solved today’s problem in an interesting, new way and there was nothing more to it than that. If that’s all it was and there was no intention to suggest that open source tools in this particular instance were any better or worse than commercial tools (ESRI tools excepted, of course, because James has explicitly stated they weren’t up to the task), well, then this particular experience is not the Rosetta Stone that can be used to decode the mysteries of when it is right to use open source GIS tools instead of commercial tools for the masses who are doing GIS.
In that case, ESRI, MapInfo Pitney Bowes, Intergraph and the like can rest easy, because they need not fear that open source GIS will blow them away.
I don’t put Manifold in the above crowd because the very low price of Manifold coupled with the very many open doors within Manifold aimed exactly at enabling people to utilize open source solutions if that’s what they want doesn’t make the use of open source GIS tools a threat to Manifold as it is a threat to, say, ESRI.
Think about it: someone attempting to do a web site to fit into an ESRI-centric workflow as James has done will be plenty tempted to avoid spending tens of thousands of dollars on ArcGIS server and SDE middleware even if it involves a fair amount of labor to configure MapServer, OpenLayers or whatever.
With Manifold it is not tens of thousands of dollars but a mere $225 for a Universal x64 Runtime, which provides not only the IMS but also direct connection to spatial DBMS and thousands of other features. Someone could merrily utilize open source DBMS like PostgreSQL/PostGIS using OGC WKB, publish out through the IMS to HTML or WMS or WFS-T and all that good stuff for a mere $225. I mean, even if you like spending your time programming there is a certain temptation when pre-packaged, fully integrated applications are thoroughly debugged and maintained for you to simply reach for the off-the-shelf item.
Which brings me to my final point: I don’t think that miscellaneous utilities or libraries like Proj 4 will either be the most widespread use of open source GIS or even the most influential: it will be fantastic packages like PostgreSQL/PostGIS and (eventually) MySQL. The data warehouse tends to drive everything before it, taking such a role of primacy that even whatever you choose for a GIS package becomes a mere accessory.
If anything, that was James’ missed opportunity to leverage the power of open source products: it was getting his data trapped in a dumb and obsolete SDE silo instead of doing the most open thing possible and utilizing PostgreSQL/PostGIS.
Note that here I am far from arguing against open source but instead praising and recommending a very strong, very effective, open source tool, PostgreSQL/PostGIS where it truly makes sense.
Dimitri:
To be honest, there are multiple points that I totally disagree with in the previous comment. However, they are simply differences of opinion of someone who feels that the ability to modify the tool he is using — both legally and technically — is important, and someone who feels that that is not important. Ignoring that particular problem at the moment, allow me to get to a question that might make a difference beyond the idealogy:
Can Manifold provide a customizable web interface which allows for the deployment of geographic vector data on top of Google Maps Satellite Imagery, Virtual Earth Satellite Imagery, Virtual Earth Hybrid imagery/maps, in the browser, such that users can: * Select a feature * Get attributes about that feature in the web interface * Download that feature in multiple formats (specifically, Atom, KML, and GML) for the different tools they might want to view it in * From the web interface, edit the attributes of the feature * From the web interface, modify the geometry of the feature
These were, as I understand it, the key features that the customer in question was interested in. The “Web browser” aspect was key: the users who were going to be viewing this are your everyday Joe Users. They were not GIS oriented at all.
Some of the users who were going to be editing this data were pretty close to this level: they know how to work in an interface like Google Earth, but they don’t have any GIS software, and have never used any.
And of course, there were ‘real’ GIS users, who had tools they needed to view the data created by the others in their tools, like ArcGIS.
OpenLayers allowed James (and myself) to build an interface which allowed for the display of data to non-GIS users, the editing of data by non-GIS users, and the use of data by GIS users.
It allowed (within the terms of use of the APIs) the use of Google Maps and Virtual Earth imagery
It allowed for the development of a custom web frontend and work flow that worked best for the users in question.
I’m willing to forget the open source zealotry for a minute if you are: The key thing was the ability to build a custom application, quickly and easily, based on commonly used web tools, that did what we needed.
Perhaps this would be simple with Manifold, I’m not sure. To be honest, I thought that Manifold was a desktop appliction: I didn’t know that it had (or could be set up as) a server to provide data to OpenLayers other than through the “IMS”, which I thought provided images, not vector data. If I’m wrong, it’s possible that a Manifold based server would have been better than FeatureServer in some cases. However, I don’t see in your licensing options what I should be buying… and that still doesn’t necessarily solve the web client problem.
If your statement is “James’s users might have been better off using GIS software to do all their GIS work, instead of working in a web browser” — you’re making a mistake on the target audience of the tool that we built. The key portion of the project was that the end result be viewable by users on the customer’s web site as a draggable map that would allow users of the site to see information about geographic features in an area (including attributes about them) as a dynamic map, without any tool other than a web browser, and that key piece of technology doesn’t seem to be present in Manifold, unless I’m missing something. Can you tell me where I’ve gone wrong?
Christopher:
I appreciate the sincerity and clarity of your posting. You make it clearer what your mission was with James so I now better understand why you went the way you did.
My thesis is about making choices and striking a balance between commercial tools and open source tools – it is not about Manifold as a specific package so it is important when Manifold is used as an example not to allow that to cloud the general conversation about balance between commerical tools and open source tools.
But I do agree that it is sometimes very useful not to keep discussions totally abstract and to call upon specific examples or counter-examples. In that spirit, I use Manifold because I am most familiar with that GIS. If someone says “oh, you can’t do that with commercial software” and I can cite a precise example of doing it in Manifold, I just do that as a concrete example and not, not in any way, suggesting that you could not also do that with other commercial tools.
So in that spirit let me respond to your specific questions:
Yes, of course. There are many ways of doing all of the above within Manifold depending upon the preferences of the web architect. Manifold, whether utilized as a desktop application, as an objects library or through the IMS objects is highly customizable and people do that sort of thing all of the time. In fact, if you view the various IMS tutorial examples (deliberately kept simple so that the particular focus of each can be understood) you’ll see examples of programming various parts of the above.
In fact, from James’ postings there is more to your task than the above and that is to achieve a dynamic connection with SDE geodatabases, which Manifold can also do.
For the sort of application you describe, you’d probably use the WFS-T (OGC feature server with transactions) capabilities of Manifold IMS. I personally think there is a better way (see below) but there are many people within the Manifold community who disagree with me and who prefer serving vectors. In fact, folks like David Brubacher (who gave a talk at the recent user gathering in Phoenix) are actually basing new products on the idea of AJAX feature serving using Manifold IMS.
Note that since Manifold can dynamically link drawings from an SDE geodatabase you could indeed create a web site that enabled clients through WFS-T to view and edit features with the post-back edits quite literally manipulating features (such as adding, deleting, editing parcels, etc.) within your SDE geodatabase.
My only caveat is that what you download those features in would be by default in one of the 80 GIS or DBMS formats such as KML, which Manifold writes to (it is immensely popular as a KML editor to create Google Earth content) but not to formats like Atom which no one uses for GIS data. It is difficult to find a serious GIS package of any kind which cannot read one of the commonly used GIS formats, such as shapefiles, so if interchange with other GIS tools is what you ask, yes, sure. I believe one of the published tutorial examples allows drawing a selection marquee and then downloading a shapefile with the selected contents. [Folks with strong tastes and programming skills could, of course, run the output of Manifold into their own tool for massaging into whatever format they liked for download.]
That’s an extremely common situation with users of Manifold. Really, you’ve described something which is typical of just about every town, county and state worldwide, where a mix of desktop GIS, enterprise GIS and IMS is required. That’s why all this stuff is in Manifold.
Just about every organization that works in a significant way with GIS data has exactly that cast of characters: a relatively small percentage of people actually “do” GIS while most members of the organization simply view the results of doing GIS. Distributed, enterprise GIS functionality on clients takes care of the first class of people while publication via IMS takes care of the latter.
By the way, even though you can use Manifold very effectively to accomplish the architecture you described, just because you can do it that way doesn’t mean it is the only way or even the best way.
Our internal research indicates Manifold in IMS uses out-installs ESRI by over 100 to 1. It is not even close. That is not a very surprising fact when you consider that someone can deploy Manifold IMS for as little as a $100 Professional Runtime license [which, by the way, includes full HTML, WMS, Image Server and WFS-T capability... and for 64-bits to boot...] so there are a heck of a lot more people who afford to run IMS with Manifold than by forking over the cost of ArcGIS Server or ArcIMS. In fact, quite a large percentage of folks who buy Manifold with no interest whatsoever in IMS end up putting up an IMS site all the same, simply because all Manifold licenses except Personal include IMS. If you spent $295 for a regular Manifold license for your desktop and you discover that you can put up an IMS set with no further purchases required, well, quite a few people do.
My point is that we have a lot of people using IMS and those folks are not at all shy about telling us what they want to do and how they want to do it. Here is some of what they tell us:
Experience shows that when people publish IMS sites for totally inexperienced colleagues, they usually find out that their audience sifts out into two classes of users:
First, those who simply want to view, get attribute information, make selections, run queries, maybe grab something off the site for download, etc. These people are mainly consumers of data sets even if they run lightweight analyses against that data and even if they are making lighweight contributions to the data set (such as, contributing point data, editing/updating attributes, etc.).
Second: Those folks who want to edit the geometry of the underlying data set. An example would be clerks in the county office who need to edit the shapes of parcels or add new parcels or to edit the boundaries of the county.
Regardless of what toolset you use it is pretty easy to satisfy the first group of users. The second group, however, quite rapidly become more sophisticated than they were at the beginning and once they get a taste of “hands on” work directly on a non-timeshared machine that’s all they want.
Once people get a taste of the incredible density of capabilities and raw speed made possible by editing “in the box” they usually don’t think much of editing through the web no matter how sophisticated the interfaces you present.
In the legacy GIS world, folks can’t afford to do that and are stuck with time-sharing (which is what your architecture is) because they can’t afford the cost of effective software to take advantage of distributed processing. But if the cost of such software is very low, people can take advantage of distributed processing.
So, although there are of course exceptions championed by folks who prefer a web-based architecture, the great majority of Manifold installations in larger organizations tend to be as follows:
An enterprise DBMS, usually a spatial DBMS of some kind, is used as the central data repository. Oracle and SQL Server are hands down the most popular choices.
Folks who create data sets and maintain them, for example, editing parcel drawings, connect to that DBMS using Enterprise edition Manifold licenses and execute Manifold on their desktops as a super-rich client to the DBMS. Thousands of people can edit the same large drawing at the same time with automatic identification and resolution of editing conflicts. The cost of Enterprise Edition in volume can be very low, under $200 a seat or even less if License Server is used so that floating licenses can assure that no license need be purchased for seats that go unused.
Manifold running as IMS sits on web servers also connected to the central DBMS to publish data to intranets and to Internet. Usually, the intranet applications allow some degree of editing or querying that is greater than the Internet applications. If people want to download data via intranet or Internet into local storage, that’s rigged into the application to allow such downloads.
Yeah, sure, people do simple editing through the web all the time. But for significant editing they tend to reach for a super-rich client (Manifold) with a direct connection to the data warehouse. It’s not because they can’t do web editing (people do that all the time with Manifold) it is because they prefer a non-timeshared, super-rich local client much better. And, by the way, the less experience people have with GIS tools the more the above scenario tends to hold true.
For that matter, many applications developers prefer the configuration of a local client (Manifold is highly customizable, allowing elimination of menus you don’t want newbies to see, allowing adding of add-ins and pre-built queries and scripts to custom toolbar buttons, etc.) to cobbling up and maintaining “editing through the web” applications.
I think a good example of the above was the presentation at a recent user conference by Robin Wood of ScanControl on his company’s tools for vineyard management, as currently used by Kendall Jackson (KJ) and many other large wineries.
When you are running 50 vineyards plus winemaking facilities spread out over hundreds of square miles you have a lot to keep track of. KJ keeps track using detailed maps of individual vineyards. The law requires them to keep track of every application of chemicals and even who did the application. KJ wants to know, with hundreds of attributes, exactly what cost and labor inputs have gone into individual, different parts of vineyards so that they can see with very fine resolution what is productive and what is not, what makes good wine and what does not work so well.
ScanControl runs everything on a SQL Server DBMS. Field workers, some of whom have not even touched electronic equipment before, carry smart phones with embedded camera, GPS and a suite of small apps. They also carry a bluetooth, pen-sized barcode scanner.
They can call up all sorts of pre-built forms, choose from pulldown menus, scan the barcodes of employees and chemicals and locations, make notes on what they do, take pictures of vines or grapes or whatever, and all that gets automatically replicated into the DBMS servers.
ScanControl used to use richer PDAs and spent a lot of effort on in-the-field editing only to discover that by far that stuff was better done in the office with field staff collecting data not by editing parcel shapes in the field but instead by using rich sets of tools to edit attributes, take pictures and the like.
In KJ and contractor offices worldwide, people have Manifold running connected to the DBMS to do all sorts of sophisticated work and queries and editing. Someone takes a picture of a sick vine in the field and an analyst sitting in France or Australia or California can click on the icon in the Manifold display to view it, and immediately send off instructions “dial back the irrigation…” or whatever.
They also publish data out to the field using IMS, so that foremen or more sophisticated users can cruise along looking at the fields around them through thematic presentations that they can call up: what needs watering, what the work ticket is for this plot, all that stuff.
Because the application is tightly integrated with SQL Server, all the rest of KJ’s management suite works as well. KJ knows the content origiun of (literally) every bottle and they know their financials (inventory, revenue from, etc.) on every bottle as well. So they can instantly create thematic renderings of any property they own to see in fine detail which vineyards are most productive or best for a particular purpose. That allows them to invest resources where they do the most good.
The above is a very typical application that really doesn’t use anything but HTML and some cell modems in the smart phones. I give it as a real-world example where less-experienced GIS data consumers usually do better when working with a simplified interface, and also to point out that you don’t always have to use a limited set of web interfaces like feature servers, it sometimes makes better sense to connect directly to DBMS with either super-rich or thin clients.
Manifold is not just a desktop application. The same package can also serve images (in a variety of ways: HTML, WMS, image server) or vectors (WFS-T).
ESRI (and much of open souce) use a non-integrated approach to web stuff: you need an editor like ArcView or ArcInfo to whip your data into shape interactively, an ArcSDE thingy to talk to DBMS, an ArcObjects thingy to program stuff and an ArcGIS Server thing to publish to the web. And all that stuff was written by different teams on different schedules using different object models.
Manifold incorporates all that into a single application. The case for doing this is at
http://www.manifold.net/info/ims_top_ten.shtml
Therefore, you buy Manifold IMS by getting any Manifold edition except Personal edition.
I’d recommend any edition from Enterprise ($395) on up. Although Professional ($295) also includes IMS, I suggest Enterprise because Enterprise vastly expands the utilization of spatial DBMS and it is only $100 more.
For deployment onto a web server you’ll need either a Professional x64 runtime at $100 per server (one price no matter how many processors, no matter how many web sites you serve, visitors, connections, etc.) or Universal x64 runtime at $225 per server. The Universal runtime is basically an Enterprise license plus Business Tools, Geocoding Tools and Surface Tools extensions. It is so inexpensive that folks pretty much automatically deploy using Universal just so they know they have it all.
There are no maintenance fees. Updates providing bug fixes are free within a given release level to all licensees. About once a year a major new release comes out (hundreds of new features and improvements) and that new product is traditionally offered to licensees of the previous release for a flat fee, usually about $50.
Well, it is only a problem if you make it a problem!
Seriously, whatever WFS-T server you use you will need a WFS-T compatible client. Use the same one with Manifold.
But better still, consider my remarks that if you genuinely need any sort of significant polygon editing you could well do much better to simply toss a full Manifold license onto the users’s machine and utilize a direct connection to the DBMS. It’s up to you either way.
Yeah, sure Manifold can do all that.
Where have you gone wrong? Well, you haven’t really, except only in the case that as technology improves we are all going to have the pleasure of seeing newer and more cost-efficient things surprise us as they pop up. That’s a good thing.
Especially at a time of rapid technological evolution we can’t all be experts on all applications, especially exceptionally broad and deep ones like Manifold that come at you with an evolutionary path of around 1000 improvements a year. Look away for a few months and it has grown more than ESRI did this last decade, so yeah, sure there will be surprises.
It’s also human nature to try to understand the world in terms of what we already know, and ESRI is not exactly eager to tell ESRI users just how far behind they are. If you have spent 20 years going to user conferences in San Diego and being told that you need to buy separate, psychotically-expensive packages like the ArcGIS suite and ArcGIS Server you would not expect that you could do much more than all that for around $500 with a single integrated package.
I had lunch with one of our phone folks today who told me about a call he had this morning. A guy had spent $180,000 on an Enterprise ArcView license and was looking to bid a job using Manifold for an IMS application. The thing that astonished this caller was that for $425 (Enterprise Geocoding Tools bundle) and another $225 for the web server he could for well under $1000 have everything he needed to develop a very rich spatial DBMS application front-ended by however rich an IMS web site he cared to create. He was surprised by that given what he had already spent on ESRI. So you are not alone in your reaction.
However, in all fairness to the mainstream, though, we don’t get that same reaction from the zillions of Microsoft mainstream people who buy Manifold. None of them have been exposed to the notion that they should have to pay tens of thousands of dollars to get an IMS site up. Tell those folks it is $225 a server for Universal x64 runtimes and they’ll ask, “Wow… that’s a lot… How do I get a better price?” … my kind of people!
Dimitri:
Just so you know, I personally give up about halfway through reading your comments most of the time. Being a bit more concise might help others like me who are lacking long attention spans to work their way through your arguments.
I’m going to try to respond where I can, but I’ll point out that FeatureServer was a handy part of the equation above, but there’s no strong reason it was chosen other than “It was there”.
The key part of the application is the web frontend (at least, as far as I’m concerned) — the FeatureServer chunk was unimportant. And it doesn’t sound to me like Manifold IMS really changes the situation (other than forcing the use of WFS-T over a RESTful web-oriented architecture for feature editing, I guess?) — there isn’t an out of the box web editing application shipping with Manifold, you have to put it together yourself. (I could be misunderstanding you: I apologize for confusing the issue with bringing up cost, which isn’t really relevant.) It sounds like what you’re saying is “Sure, you can use OpenLayers with Manifold, but you’ll have to write that frontend” — in which case, Manifold wouldn’t solve any real problems either way.
There are examples of how to do it: great. FeatureServer comes with an example of how to deploy it, along with OpenLayers, to edit features, though the easiest way to deploy it (If you have the most widely targeted setup) is to simply unpack the tarball in your webserver directory and visit the directory in your web browser, since it ’ships’ with an OpenLayers-based example for editing features and attributes pre-configured.
There are cases where running Manifold for this would make sense. I can’t speak entirely to the cases where it does or doesn’t. However, this problem was pretty simple from the point of view of the server — we grabbed FeatureServer because it ended up doing what we needed, and being trivial to integrate with OpenLayers. (Does the Manifold gallery have OpenLayers examples? If not, I’d recommend ‘em. I’d check myself, but I couldn’t find it, though I’ll admit that if it wasn’t attached, I often wouldn’t be able to find my own head.)
Building the client seems to me to be a one-off task either way, and that one-off task is the one that mattered. Although James concentrated here on the fact that FeatureServer already did what he wanted — mostly because that’s the impressive part — the part that really mattered was the fact that it went into OpenLayers trivially, since that’s the ‘hard’ part of this particular problem.
You mention “You’ll need a WFS-T client either way”… but we’re not working with WFS-T here, we’re working with a totally different type of editing. I’m not aware of any good web-browser based WFS-T editors. OpenLayers isn’t one (though you could make it be one), primarily because WFS-T is ill suited for web-based editing.
Does the Manifold IMS ship with a web-based WFS-T client out of the box, with support for editing on top of Virtual Earth and Google Maps? If so, is it really a key part of Manifold, or could we seduce you into sharing with OpenLayers and helping improve web-based editing for everyone? I’ve limited experience with web-based editing outside of my narrow open source view: in general, I’ve not been impressed, but maybe I’m looking in the wrong direction. Still, if Manifold is maintaining its own web-based editing toolchain, it might be able to benefit from OpenLayers; I can go into that more if you’re interested in how it could help you (even if you don’t want to help OpenLayers out). The license (which I’m not sure why you feel it’s hard to find: it does say on the homepage of the project “Released under the BSD license”) makes it perfectly easy for you to steal the code and offer nothing other than a line that says “Includes OpenLayers” in your documentation.
Anyway, I was just a contract guy. I was asked if I could do a job, and how much I could do it for. I quoted a number, and an implementation. It’s the only one I can put together: I don’t have access to any computers I can install Manifold on. OpenLayers and FeatureServer both worked great. They both got some new features — which are now fully supported, and no one will have to pay for them in the future, if they want them. The OpenLayers community also benefits, and extended some of the work I did even farther, so the limited amount of money that was invested paid out more than it might have otherwise.
Although I appreciate your concentration on the area you know best, and I agree that I’m doing the same, it sounds to me like the tools we used weren’t the wrong ones (though there are obviously alternatives). Which brings the entire disagreement down (in my mind) to being over the value of Open Source: with that in mind, it’s probably best to declare it a truce and move on, since using Open Source software has a significant benefit to its users that is in the ideology, and if you’re not convinced, there’s no need to try to become convinced (as far as I’m concerned, anyway).
hello,
I just wrote this article on a related subject:
http://zeidnertechshowcase.blogspot.com/2008/02/data-security-and-google-member-maps.html
I’m hesitant to add to this lengthy blog but here it goes. My question, or argument, is this: where does the cost associated with the learning curve come into play? As far as I can tell all of the open source solutions I have seen are pretty much figure out how to use it and implement it on your own. They may have a website with some minimal documentation, but I have not seen much instruction on how to use a particular solution. I know there are smart programmers out there that can figure things out, but shouldn’t that be included in the cost of implementing such a solution? Say what you want about ESRI but there is a wealth of documentation, books, courses, and support.
My company is evaluating different solutions for evaluating different map serving/enterprise GIS technologies and currently we are struggling with the cost of learning to use the different solutions. This also leads to my next question: as a consultant how do you deliver a solution based on various toolkits/APIs that the client is unfamiliar with? Are they comfortable getting something that no body in their organization has experience with? What happens when it breaks or they have to modify it in someway? Do they have to call you for support? Maybe that’s the beauty of it.
Don’t get me wrong I am not arguing for one way or another. I am just looking for answers to my questions.
I agree with you 100% that ideology is why some people, perhaps even many people, pursue open source solutions. That is exactly one of the points I set forth in that famous letter to the editor.
But as much as I agree with you that a key benefit gained by open source users is identification with a particular ideology, that is a psychological and social matter, not a case for operational efficiency. For that matter, I think acceptance of ideology without a sound, pragmatic case for why that ideology performs in real life better than alternatives is the way one gets into dead ends. Launching into a life based upon ideological slogans like “From each according to his ability and to each according to his need” without first considering whether it was possible to map such slogans into practical reality is exactly how the beloved homeland of my people ended up enduring 70 years of communist ruin.
I say that if you are a true friend of open source you reject mere ideology and you insist on hard headed arguments that in every detail make a common sense case why your approach is significantly better. That is the way to make open source stronger, because it rejects the slothful, slacker path of confusing slogans for competitive value. If you don’t have a significant reason for being beyond the appeal of a stylish, but squishy, ideology you end up getting marginalized by other approaches which do have enduring, real value.
Well, sure I could be more concise if I didn’t have to lay out an argument on your side as well as mine!
Seriously, getting open source folks to explain exactly why what they are advocating is better than a commercial approach is like nailing jelly to a tree. The open source guy lobs up some jelly in the form of a statement like…
…and then one has to spend endless paragraphs unfolding exactly what the above means, trying to nail that jelly to the tree. I mean, if I am to respect the intelligence of readers I can’t just reply or “oh, there’s no zero overhead” or “oh, you’re just assuming your conclusion” because then it is not a mature discussion but rather a decomposition into an exchange of sound bites.
Speaking of “assuming your conclusion” I think you are doing that here. Consider:
Pausing a moment to note that the out of the box editing application is Manifold itself, let me now get into why the above is “assuming your conclusion.”
If I understood the description of the client correctly, they wanted to be able to edit features in an SDE geodatabase and to do so on the desktop as well as on remote desktops, to be able to publish their data for viewing to a larger audience on a web site and also to be able to download selected parts of the data set for local editing by local tools. That is the task.
If your clients are indeed non-GIS people, then I’m sure they did not come to you by saying “I want a RESTful web-oriented architecture”. They simply said “I want to have remote access to my data, to see what it contains, and to edit it.”
I agree that when someone comes and says something like the above to a consultant who is predisposed to “open source” (whatever that is) as a matter of ideology it is often the case the consultant will hear only what he wants to hear. If he is driven by ideology rather than optimal solutions, he may well indeed hear the client say “I want to use only open source tools, it cannot have anything to do with Microsoft, it must be web centric and it cannot have anything to do with commercial companies except Google and SUN and Red Hat, which I think are cool, and it must interoperate with stylish but dumb standards that no one in the mainstream uses, like OGC things, for example GML.” And then when you ask the consultant why he did what he did he replies, honestly, “that’s what the client required” when in contrast it was not what the client asked but rather only what his ideology allowed him to consider.
Let’s take the present case as an example. You haven’t told us significant details of who the client is or what their application and needs as an organization really are. But I have gathered that it is a sophisticated, valuable set of geospatial data (people don’t undertake the cost of SDE for trival things) and that there is a need for users in various locations to be able to edit that data.
Let us zero in on this task of remote users editing data, since I don’t think there is any controversy that all the rest is easily accomplished with Manifold “out of the box.”
You don’t say, but I will make the inference, that the number of people who will be allowed to edit features in any significant way are relatively much smaller than the number of people who need to simply view the results of such editing or to query or to select or to report attributes from the results of such editing.
I think that is a fair inference because no one in their right minds allows large numbers of inept people to edit features (such as real estate parcel outlines) within really valuable data. The people who will be doing significant editing are serious folks who deserve serious editing tools, not barren junk.
The solution I propose for that is not a Rube Goldberg assembly of a set of intermediaries, it is a direct server – client relationship between the DBMS that stores the data and the client that people use to work with that data. To answer your question about clients, in this case Manifold itself is the client.
As far as I can tell, your solution is overly modular and indirect in order to solve problems that don’t really exist. Let me give a technical re-statement of that:
Your solution has a cascade of players (I will paraphrase):
The DBMS exposes interfaces. SDE interacts with those interfaces to expose an SDE geodatabase API interface. FeatureServer interacts with that to expose an interface that, across the Internet, is consumed by OpenLayers working together with your browser. Your actual edits occur in a fairly large local client, the browser plus OpenLayers. That’s an awful lot of stuff between the DBMS where the data is kept and the client sitting on the desktop that actually provides a GUI/commands/capabilities to edit that data. Every bit of that stuff can break in a nanosecond when any one of the bits and pieces you have utilized changes slightly under the influence of whatever ruling group administers it.
But you are not alone in your approach. A lot of open source architects take a similar approach, more typically:
DBMS … WFS-capableIMS … WFS-T over Internet … Client
Ultimately, you end up using intermediate machinery and protocols to have a client sitting locally that can provide whatever feature editing capabilities you need. However you get that client, it will be significant software if you wish significant editing capabilities.
What ends up happening if you are not willing to settle for a very barren, limited, low performance client experience is that you end up having to write code to create richer and richer clients and then figuring how to service that client through the bottleneck of your feature server and the feature serving protocol (WFS or whatever) you prefer.
Although you can take that approach with Manifold (I used WFS-T as an example because it is an Open Geospatial Consortium, indisputably “open” thing…), a better approach would be to cut out all the Rube Goldberg stuff in between the DBMS and the wonderfully rich feature editing capabilities of the client:
DBMS … Client
In the above case the DBMS is whatever you want it to be and the client is Manifold itself, communicating directly with each other through the magic of TCP/IP, VPN or whatever high performance pipe you care to run through whatever WAN you are using.
You don’t need no stinkin’ middleware to slow things down and to deny ultra-rich clients the benefits of ultra-rich DBMS capability, and you don’t need no stinkin’ abstractions designed for nothing more but “interoperability” to software no one uses. The DBMS you choose in and of itself defines an “open” ecosystem so you may as well speak to it directly. In my recommendation, Manifold as a super-rich client, out of the box, can connect through the web directly to that supremely competent DBMS you use.
This also has the benefit of distributing intelligence, and thus performance, out into your clients. Want Google Earth or Virtual Earth layers in the client? Sure, no problem: the client itself gets those layers directly. Want an astonishing array of user-friendly editing capabilities within a wonderfully supportive GUI? All there.
No, you are not looking in the wrong direction: you see unimpressive web-based editing because web editing is an attempt to bring back time sharing from the dead and so from a performance and richness of capabilities perspective is totally lame compared to exploiting the power of millions of distributed desktops.
Whether you call that exploitation “Ajax” or business as usual, it is ultimately the same reason that time-shared applications got blown away by people using many, local processors.
In modern times, it is people who are expensive and processors that are cheap. To time-share a web server processor among many people is stupid, stupid, stupid – instead you should be deploying many processors, at least four and ideally hundreds (the CUDA thing) of processors for each person so that a vast array of processor power slaves away every microsecond just to make every little and large thing easy for the user. Throw away a trillion cycles so that some mouse move becomes a bit more effortless for the user? You bet.
For a detailed explication of the above, including gratuitous jabs at morons who don’t understand what “Ajax” means, see the essay at
http://www.manifold.net/doc/what_about_ajax_.htm
…. which manages to state a variety of enduring technical truths in a marvelously incendiary way.
I’m all for a truce with open source folks because I happen to enjoy open source. I just hate to see it devalued by people who don’t themselves know what they mean by that as free of cost or not free, free to modify or not free, as a commercial product or not a commercial product or who are not able to articulate why it makes sense to use a particular package in a given case.
Over the years, I’ve put my money where my mouth is. To date, the single most identifiable corpus of “open source” stuff in GIS has been the work of OGC (formerly the “Open GIS Consortium” and now renamed the “Open Geospatial Consortium). Manifold has distributed by far, certainly by an order of magnitude and perhaps by two orders of magnitude, more software to the masses that enables utilization of OGC “open” standards like WKB, WKT, WMS, WFS-T and even GML. It has by a wide margin been the only mainstream GIS package to support PostgreSQL/PostGIS spatial DBMS, and it was the first to open doors for a wide variety of “open” ideas, from truly open image servers like WorldWind, to quasi-open but nonetheless fellow-traveller ideas like using EPSG and XML to support vendor-neutral projection system specifications even within deeply proprietary systems such as Microsoft’s new “Katmai” spatial DBMS. For that matter, Manifold has even directly participated in “open” programs such as by contributing engineering staff to correct errors and help create the open source distribution for Windows of things like ECW and JPEG 2000, a very useful tool against outright proprietarizations like MrSID. And, we even indulge in our own sanctimonious “open” snits from time to time like the various corporate rants against GIF, against governments denying public access to public data, our famous litigation to crack open TIGER/Line and VMAP 1 for free public access and the like.
So, no I have no problems supporting and even encouraging open source. I just wish that the ideology of some open source folks allowed them to be equally flexible in considering the merits of using commercial products within Windows systems as well!
Dimitri:
You don’t want to be convinced. I don’t want to convince you. Together, we have come to a position that we can both agree on: You have your opinions, I disagree with them, and everyone can continue business as usual.
Luckily, I was never trying to convince you: I was trying to understand what you thought Manifold did better. Since I can’t run Manifold, I can’t develop for or around Manifold, and since I can’t develop around Manifold, it’s not an option for the solutions I propose at this time. I was trying to figure out if I should change that: The answer appears to be ‘No’, based on what you’ve described.
I appreciate the new things I’ve learned, and I’m confident that for most of the problems I need to solve, I have the right stack of software to solve them, and that Manifold isn’t in it. I appreciate you taking the time to help me understand that; it’s always good to understand the solutions you don’t choose as much as the solutions you do.
I hate to distract from the discussion, and my comment isn’t pertinent to the discussion, so please forgive my intrusion, and obsession with imagery. But, has anyone else noticed the three images that James gave, as examples, are not the same image?
I just thought that it was interesting to see three images of the same location that are not the same image. It shows how easy it is to obtain imagery now.
OK, back to lurking in the shadows.
Christopher:
I apologize if I gave the wrong impression. The above is not correct.
Yes, a solution using Manifold plus your favorite DBMS will perfectly and beautifully fulfill your customer’s requirements.
It will indeed enable both local and remote dynamic editing of features within the ArcSDE geodatabase in highly sophisticated ways using a wealth of editing tools within an easy-to-use GUI. It will allow publishing those results for the masses through a GIS-enabled web site and it will enable people to be able to visually select data of interest and download it within a variety of well-known GIS formats for use locally.
There are several different ways the above could be implemented, ranging from almost entirely out-of-the box, high performance solutions (which I recommend) to various other architectural approaches some experts may prefer better (which would be similar to your proposed architecture).
It is true the above is not the same as your chosen FeatureServer OpenLayers project and it is true that if you are committed to specific software product, such as OpenLayers, Manifold may or may not be able to play a role in your implementation. If you already have a solution that you have created with which you are happy I agree it would not make sense for you to toss out what you have done and to re-implement it using Manifold. I did not mean to give the impression that was my intent.
Likewise, although I gave examples using the toolset with which I am most familiar I don’t mean to imply that this could not be accomplished using other commercial GIS tools. No doubt people who are experts in ESRI, MapInfo Pitney Bowes or other commercial systems would be able to offer their own examples of how your customer’s needs could be accomplished using the tools they know and love. I’d be astonished if the ESRI folks could not present a completely ESRI solution, for example.
I am mildly disappointed you would not want to pursue the discussion of why web-feature editing is such a terrible idea compared to rich clients connecting directly to rich DBMS using the power of distributed processing. It is not something to lightly discount, as a modern DBMS-centric approach can yield astonishing gains in speed. If someone can use distributed processing to achieve 100-fold faster performance, they get the business.
But perhaps that’s a different debate that perhaps merits its own thread in a different forum!
Dimitri:
You seem to think that the customers described in this post are “My customers” — but they’re not. The first sentence of your statement is, in fact, the problem that I have: my customers can’t install anything without a 2 year lead time. (2 is if you get on the ‘fast track’ for a new application: usually it takes significantly longer.) Since I usually work on a project for a maximum of six months, using Manifold as a solution doesn’t work.
I was mostly curious as to whether Manifold as an IMS made more sense than FeatureServer, and it seems to me that for the specific problems (not described here) that I’m trying to solve, this isn’t true. That was the knowledge I was able to gain via our discussion, and I thank you again for it.
Christopher,
Wow! That’s a tough set of customers to work with!
A two year lead time means they are always working with ancient history.
Still not sure why you couldn’t use OpenLayers as a client to Manifold IMS being used as a WFS server. I’ve visited the OpenLayers.org site and it seems any of those examples could work just fine with Manifold. Tell you what, though, just as an intellectual curiosity I’ll ask around and see if anyone has done this. When I find someone who has, I’ll report back what they did, what they liked and disliked about the Manifold + OpenLayers combo, etc.
Dimitri,
I do enjoy the articles on manifold’s website. I also find it funny that one of the glaring omissions is the fact that remotely using manifold or using it locally (with data from a RDBMS) requires that same type of SLOW network connection that is required of web applications. While it might be mitigated somewhat by local network speeds, for remote users, the same sorts of speed concerns do apply.
The 2 edged sword of network/internet speed (or lack thereof) then cuts against both local applications and web applications.
For me, the real limitation of the web application is brought about by the use of the browser in which the web application runs. I can create a GIS web application (and I have) that handles data issues in ways savvy of the network bottleneck by caching data locally and updating behind the scenes. The issue that invariably arises is the use of the browser.
This straight jacket would at least be less frustrating and easier to deal with if there was a true standardization across browser flavors. Add to it the maddening variations between browsers and we are left with a better understanding as to the pain it is to develop web applications.
AJAX can and is a good thing (if used correctly and in complete understanding of its limitations and benefits) because it helps to alleviate some of the problems that deploying a web application into a browser presents.
I have recently begun to experiment with direct web application to DBMS interaction via native spatial data types. This is another means by which we will be able to enrich the web application user experience.
Finally, I think it wrong to place both web applications and desktop applications into a direct competition. Each of course have advantages and disadvantages. There are some very real benefits to running web based applications for certain kinds of problems (even some problems which are normally thought of as traditionally desktop applications). One example is the ability to integrate the GIS Application Server/Web application into automated business process tasks. As said in the articles on manifold’s site, human time is at a premium and the ability to throw cpu cycles (and 24 hour processing) at problems make web/network based GIS automation a viable and profitable solution.
Being unfamiliar with manifold means that I have most certainly not understood everything at the fingertips of its users so I do not mean to ‘knock’ this application. But I also want to make it clear that to knock GIS over the Internet using the arguments presented (based mostly on the 2 papers on the manifold site) is not telling the whole picture.
Thanks, -OC
Dimitri:
Sure. I’ve seen that Manifold can possibly do what FeatureServer + PostGIS can do — but not why I would prefer it, when integrating Manifold into OpenLayers is likely harder.
@Christopher
“my customers can’t install anything without a 2 year lead time. (2 is if you get on the ‘fast track’ for a new application: usually it takes significantly longer.) “
I take it, the customers are the Feds? Sure sounds like my outfit. They get in “new” software, go through a certification process and probably a year or two it’s avaliable.
Of course, like Dimitri said, it’s behind the curve once it’s released to the rest of us.
KoS
KoS: The feds is a fine guess. It’s pretty much no different in any of the enterprise markets we (MetaCarta) target: Even for upgrades of our product, some customers ask us to limit the number of upgrades to one a year, because getting them through the approval process to install even on the appliances we shipped is hard.
Enterprise markets are painful to work in. Web-based applications make life easier, since all computers have a browser, so people can use the tools we write without the many month lead time. Since they’re not GIS people in general, there’s almost no likelihood of getting GIS applications (be they open source, closed source, ESRI, Manifold, or otherwise) installed on the actual desktops, so we resort to the lowest common denominator, and hit a much wider audience.
I am tired of seeing Dimitri trying to convince everybody about how good Manifold is. It might be, but I guess this blog is not for selling Manifold. I prefer to see this happening in a blog dedicated for Manifold. Otherwise rename ’spatiallyadjusted’ to ‘manifoldadjusted’.
I hope also he takes my message in good faith and not reply to me with an endless post in which he will try to convince us he is not selling Manifold or whatever…
Vasilis:
I’m not sure that Dimitri is doing anything different than what I’m doing: I try, most of the time, to make a strong case for OpenLayers, FeatureServer, TileCache, etc. as tools to use when your needs require web-based applications to be deployed to a wide variety of users. Dimitri is doing the same, for a different use case. With that in mind, how is Dimitri any different than I am… or perhaps you think that I’m also at fault, for advising people that OpenLayers may be a better solution to their problem?
hmm I see your point. But in your case you talk for an open source project in a thread titled ‘Bringing Open source in to ESRI Shop’. Making so much noise for Manifold in such a thread I guess shifts away from the central point of the thread. That’s all.
plus your posts are clear and concise
My comments on OpenLayers aren’t limited to this thread: they span most of my commentary on blogs across the space in question.
You can personally disagree with Dimitri’s comments, but that doesn’t make them wrong; he’s just taking a different perspective than some of us are.
So long as people are pushing Open Source, Web Apps, non-Windows development, etc. one can expect that Dimitri will continue to point out that he believes he has a better way of doing things. I think that’s his prerogative, even if I disagree.
Brian: I don’t think I’m particularly clear; my concise statements are usually the ones with the least content
Thank you James for hosting [honest] discussion on such a relevant and timely topic.
We are assessing third party tools to support developing a (fairly simple) browser-based mapping application; we do not want the maintenance overhead of installing client side software (or data).
Although partnering with ESRI offers the benefit of appeal to the “ESRI shop-pers”, I find the company more rigid today and than it was 13 years ago: AGS is incredibly expensive; they will not allow partners to re-sell the software; and from everything I’ve read, it sounds as frustrating as ArcIMS (brittle and not well documented).
Although Google Maps is really cool, the api is not separable from the served data; our clients require primary access to internally maintained GIS data.
Mapguide OS seems like the answer. From what I’ve read, the only difference from the Enterprise version is $59.95 and tech support from AutoDesk. I think they are making their money on the authoring tool: $495. The license is clear and supports developers that want to add value.
Please share your thoughts…TIA!!
OrganizedChaos,
Tha’ts a particularly astute post: You are right, it’s not something that always cuts the same way, and you are also right you can’t dodge fundamental limitations such as bandwidth, speed of light, etc. No perpetual motion machines here or anywhere.
The key is how you use that limited bandwidth. If you can create an application that tosses a relatively small amount of data between the server and a rich client you have the classic Ajax opportunity to leverage local processing while the relatively small amount of data being exchanged doesn’t cause such noticeable delays.
To the extent you can get a fatter pipe and reduce the traffic on that pipe, you can expand the possibility of running things effectively within remote applications using Ajax-oriented rich clients. Since you are probably already running the fattest pipe you can, you need to focus upon getting best use out of that pipe. There are some tricks you can apply.
The first is not to waste any bandwidth on infrastructure and inefficient protocols. Texty protocols like HTML or KML waste bandwidth. Infrastructure placed in between the storage repository and client necessarily interpose overhead, having the same effect as narrowing bandwidth in terms of response.
A direct relationship between client and data repository using the fastest possible protocols, which is what you get when you use WAN for direct connections, helps minimize infrastructure and makes the best use of what bandwidth you have. But even so whether that makes sense depends upon the application and the synchronicity or lack thereof you seek.
For example, if you are doing an application like ScanControl’s that runs in smart phones (believe it or not actually tossing the CE version of SQL Server into those phones) and depends upon SQL Server replication every now and then you will have a noticeable delay during replication/sync. But what you pick up for that is instantaneous operation otherwise, as though there were no network. Use Manifold as a client in a direct to DBMS connection and depending upon how you architect it you could also encounter a longer response time with some operations, for example, if in the remote station you are doing a lengthy spatial SQL analysis that involves the entire DBMS. That’s atypical for such applications but that it occurs shows the proof of what OC writes.
In fact, if you are creating an application where remote users always do things that involve the entire database you don’t have as many opportunities to gain by doing the Ajax thing, because ultimately you are going to be doing the heavy lifting server-side (since a characteristic of DBMS-wide data operations is that they are usually dominated by data access time, not computation) and won’t be tossing anything client-side except, what? a few spatial SQL strings? You could do that just as easily tossing bit maps like the way Remote Desktop works.
This raises the issue of not robbing precious bandwidth by tossing trillions of bytes back and forth to do what could be done locally. Very rich commands involve a lot of interactivity between the data and the GUI. If you are running an Ajax-style distributed application, that density of interactivity can happen between the command set and what is, in effect, a local cache. So it does not use network bandwidth at all and thus runs with the speed of an application distributed into running in local processor and RAM. That’s what Google Earth does to outstanding effect.
On the other hand, if you try to implement a sophisticated command set using a super-thin client, say, like just an Internet browser, with all command and GUI intelligence happening server-side, you will potentially use a lot more bandwidth to communicate user interactivity to and from the server.
[I grant you that a good design can minimize the amount of extra bandwidth that is required. But still (jumping off the thread about bandwidth) you end up timesharing the server.]
That, too, is a very accurate and astute observation. People forget that browsers tend to be relatively inept hosts for serious processing. Try to write a plug-in to IE or FireFox that enables you to duplicate PhotoShop functionality and you will see what I mean.
Manifold has done plug-ins for browsers, like the Manifold Toolbar:
http://www.manifold.net/toolbar
This was an interesting exercise because implementing the toolbar showed how browsers are not a very good environment for hosting very high performance graphics.
[It is also very interesting because the Toolbar is a proof by example that the attempt to proprietarize public web services like the Google Earth image server are simply efforts by Google to force you to use their browser and only their browser. If Microsoft tried to offer a public web site that could only be visited by Internet Explorer you'd have half the planet (well, the European Community at least...) heading off to Redmond with their pitchforks and torches ready to burn the place down...]
But, I don’t think the browser plug-in notion is the weakest link in something like OpenLayers. The ability to use a browser is a strong point for OpenLayers and the relative performance loss by working within a browser probably is not as significant in most applications as the performance loss of giving up on the idea of truly Ajax-like, distributed superclients.
OrganizedChaos:
To split into two posts, since this second response takes up a different aspect of what you so astutely posted…
You raise an interesting point about browsers and software that seems to indicate a logical hole in the demands that Christopher’s customers torture him with, this two year validation stuff:
That only makes sense to me if the customer doesn’t realize that he is cheating his own two-year criterion. Like what, it is not OK to install new software (let’s say it is written in javascript, just to make the comparison equal) if it installs via a Microsoft .msi installation but it is OK to install new software if it installs via some stealthily downloaded javascript download or if it is stealthily unpacked by the consultant without the customer being told new software has been installed?
If you are saying that OpenLayers is the only bit that is being installed and that it has been vetted, well, that was an awfully fast two year turnaround on OpenLayers!
For that matter, OpenLayers is not the only software involved. There is mucho server-side and middleware stuff going on. Do people trust that without validating it?
Setting aside the fact that new software is being installed, to work together using the browser, it seems to defeat the point of validating new software by suggesting that by running the software in a physically different location does not require validating it while running the software on your own premises requires validating it. If anything, it seems to me that someone would want to be more careful validating software controlled offsite than software which is entirely under his or her physical control.
To see how the above plays, consider the following cases:
Case 1: John installs a software package that runs entirely in his computer.
Case 2: John installs a software package that consists of a client and a server, but both the client and server are installed and run entirely in his computer. John’s client is a browser augmented by some custom code that provides the desired client-side functionality.
Case 3: Like Case 2, but John installs the server software on a second machine sitting in his neighbor Frank’s cubicle. The two machines communicate on the LAN across the four or five feet that separate them.
Case 4: Like Case 3, but instead of the server sitting on Frank’s machine in the adjacent cubicle, the server machine sits in the organization’s IT department, which could be in the next building or the next state.
Case 5: Like Case 4, but instead of the server sitting in the organization’s IT department it is sitting… where? in some third party’s IT department? …and, it runs some mix of software not validated by the organization and not controlled by the organization.
Consider the above cases and you’ll see that the idea that an organization wants to spend two years validating software in Cases 1 through 4 but bends over at the pleasure of a third party in Case 5 without validation seems to indicate a deeply retarded and dysfunctional organization. That’s not typical in GIS circles, as much as we all have our minor complaints from time to time with our organizational bureaucracies!
I know Christopher has to do his best to bear the demands of this particular customer and to work within the customer’s rules. I’m not quibbling in any way that Christopher’s solution, if it is palatable to them, of going web-based to exploit a logical loophole in the customer’s overly-bleak two year validation process is not a sensible idea. If it does the trick you bet I’d do the same thing.
All I am pointing out is that this is a fairly unique situation. In most cases customers realize perfectly well that the physical location of the software they use doesn’t make it any more or less trustworthy, and that outsourcing trust to whatever it is your browser can visit does not reduce the need to vet the software you use.
Vasilis,
It is really impolite of you and to ignore my repeated citations that I use Manifold as counter examples because that is the software I know. For example:
What I find astonishing in its absence, an absence you could remedy, is the lack of ESRI-based examples where the use of commercial tools has strong benefits over the use of open source tools in this particular case.
Frankly, I am just astonished that in a forum where presumably a lot of really expert ESRI users visit there has not been an expert response from the ESRI side that would point out the many benefits (I almost wrote “manifold benefits” but I don’t want to tick Vasilis off any more…
) of using ESRI software in such situations.
C’mon guys, you’ve spent a lot of money on ESRI software, this is an ESRI forum, you have a lot of expertise, and you and I both know that ESRI software, for all my occasional carping about the legacy nature of it, is nonetheless extremely sophisticated, supremely powerful stuff in the right hands.
If it really is the case that someone can string together completely free items with “zero overhead” as James so emphatically put it, to eliminate the need for any ESRI products without losing the many benefits such ESRI products bestow, well now is the time for everyone to clear out their cubicles in Redlands and for James to strike the “ESRI” portions of this blog because open source has made them irrelevant.
I don’t believe that’s the case, but I would very much like to hear from ESRI folks just what they see as the holes in this “open stuff succeeds bravely where ESRI does not” narrative.
How is this an ESRI forum?
“Blogging GIS, Google Earth, Virtual Earth and Programming” Doesn’t say “ESRI” to me.
Dimitri:
I don’t claim my customers are rational or logical, but there is a major difference you’re ignoring here: So long as you don’t install browser ‘plugins’ of any kind (which also fall under the ‘2-year’ general principle), running a web-app can’t break the local computer.
Now, there are exceptions to this: IE and Firefox both suffer from security holes that have made it possible to crash them, and I know that IE has had a number of bugs that might possibly make it possible to have some greater level of control over the computer, but these are of course possible with any software. By design, however, it’s not possible for software in the browser to affect other aspects of the computing environment.
Manifold could format your hard drive when you installed it. OpenLayers could not format your hard drive no matter how hard it tried.
They don’t care about protecting the server that the data is on: this is about creating a stable computing environment which isn’t changing. In that regard, none of the other cases you mentioned above has the same effect: any one of them can screw up the computer it’s installed onto. The only way to prevent that is to not install any software. OpenLayers isn’t installing software, and that’s what matters.
I’m surprised you’ve seriously never run into this: I’m surprised there are enterprise IT departments that don’t have a similar policy in place. I’ve never heard of someone who worked in a place with more than a thousand employees which didn’t take pretty rigorous standards for what employees could install on their workstations.
Dimitri: Why does anyone care if an ESRI or Manifold product could do the same thing in this context? It isn’t my job to highlight either, just describing meeting the goals of a client. It doesn’t matter if I could have done everything with either product as the client didn’t want that sort of solution. This blog isn’t devoted to pushing any one technology over another. I admit that I haven’t a clue how to develop with Java, but beyond that I’m open to any technology that my clients want. If a day comes where one wants a Manifold solution I’ll be happy to blog about it then. It is your job to get Manifold into DoD and federal government circles, not mine.
I don’t get paid to push ESRI, Microsoft, Google, Intergraph, etc solutions. I get paid by meeting my clients goals on their projects.
I promised to report what I found out about using OpenLayers with a Manifold WFS-T server instead of FeatureServer.
First, the WMS case because that is something I quickly got working and because it educated me a bit about OpenLayers:
It turns out you can use OpenLayers directly with Manifold using Manifold as a WMS server if you adjust your OpenLayers installation a bit to work around a minor defect within OpenLayers.
The WMS code that is in OpenLayers is pretty strange in that instead of asking the server what image types this server supports (which is what the WMS spec says a client is supposed to do), the code simply requests image/jpeg, image/gif or image/png, depending entirely upon its internal chain of thought which has nothing to do with what the server supports or what the WMS standard says. The result is that by default the code simply fails when connecting to a server which only supports image/png, as the Manifold WMS server does.
Thankfully, this is easily resolved by setting the image type to image/png when creating the relevant layer object exposed by the OpenLayers API. A one-liner once you know about this minor error within OpenLayers.
The situation with WFS-T is more complex. It should be that plugging Manifold’s WFS-T capabilities into OpenLayers would be a simple matter of changing the URL for the WFS server in use. That should work, for instance, in one of the published OpenLayers examples such as:
http://www.openlayers.org/dev/examples/wfs-t.html
This would include full editing – instant plug and play. However, simply changing the URL does not work. I am now investigating why, and have asked for help from a friend who is a very expert engineer.
So far, we cannot exclude a bug in Manifold WFS-T serving, although that is always possible. If a bug is found it will get reported and fixed in the next update (comes out every month or so).
However, preliminary indications indicate it is something within OpenLayers. We suspect a very minor configuration thing like the erratum with OpenLayers WMS connection. Our guess is that if the code within OpenLayers is any good (as we expect it is), the amount of tweaking required would be minimal.
Whatever we find, we will report back to the OpenLayers.org group so any corrections we create can be contributed back into the product for everyone’s benefit.
I have to wonder if maybe we are not the first to discover that OpenLayers may have issues with WFS-T connections (if what we suspect is true). Given Christopher’s comment that FeatureServer does not really support WFS (see below) as well as Christopher’s comments much earlier in this thread denigrating WFS, well I would not be surprised to learn if he had encountered a bad experience trying to use OpenLayers with some other WFS servers. If OpenLayers doesn’t bother to check WMS capabilities, I would not at all be surprised if it doesn’t do the requisite WFS checks either, especially since FeatureServer doesn’t support WFS getCapabilities() [both OpenLayers and FeatureServer came out of MetaCarta]. But it could be a bug in the Manifold WFS server so let’s not reach any conclusion until the engineering process finds the problem.
No problem either way for real men with moderate programming skills: It would also be easy to avoid using WFS-T and just use the Vector layer directly. This would require adding some code on the server side controlled by Manifold, but that code would be no more complex than the code one would have to write to use OpenLayers in the first place, such as the code that James , Christopher and other folks have to write in any event when using OpenLayers.
The amount of code in published OpenLayers.org examples is in the range of a hundred, sometimes a couple hundred lines. That would easily suffice for what one would have to write to use Manifold as well.
Based upon the above I guess that in James’s case, using OpenLayers with Manifold would be as easy as using OpenLayers with a combination of FeatureServer and OGR (which is what James likely used to store data since FeatureServer does not itself store data).
I would respectfully point out that this combination of Manifold and OpenLayers would also provide a huge missing piece in the FeatureServer implementation, in that it would supply the missing step of writing to SDE, providing dynamic editing within the SDE geodatabase, (which, to the best of my knowledge OGR can not yet do).
In the best case, once we figure out what the missing secret handshake is, you could accomplish the connection from Manifold WFS-T to OpenLayers as easy as borrowing the above WFS-T example from the OpenLayers.org site and changing a line or two.
I cannot resist a few more observations learned from the above:
First, OpenLayers is a truly charming application. I can see why Christopher is proud of his work with it, as OpenLayers has a lot of appeal. In fact, it has so much appeal that I have managed to convince manifold.net that whatever my little project learns should be supported by the company in the form of a “how to” example using OpenLayers with Manifold, both via WMS and via WFS-T, and that the company should allocate some engineering resources to package up any corrections of errata we develop to send off to the OpenLayers folks.
Second, for all that OpenLayers is quite rough, very much a work in progress. It is true that the OpenLayers.org site warns about the effectively beta nature of this code, but still it is rough to work with for integration because it is clearly a moving target.
Third, that even for experts (I’m referring to my expert friend, who is one of the best software engineers I have met in many decades), there is no such thing as “zero overhead” when playing the role of system integrator / software developer trying to bring together different modules like these. It is incredibly wasteful of time.
Finally, as part of this investigation I looked at FeatureServer, and also at the link cited by James in his other blog posting reporting Paolo Corti’s introduction to using FeatureServer. That was cause for much dismay, as FeatureServer is all sorts of jelly being tossed at a tree. For example, it is inaccurately reported as being a
Not remotely true, comrades, because as Paolo reports:
For those of a non-technical bent, that means that FeatureServer is not, repeat not, a WFS server. Try to use it with a WFS client that conforms to the WFS standard and it ain’t going to work. [Those of a technical bent understand immediately that without getCapabilities() you don't have a WFS server.] I’m not saying it isn’t a server of sorts, but the point of having identifiable standards like WFS is to avoid the need to use telepathy or painful engineering trial and error to figure out when something does or does not conform to a claimed standard.
That, along with a variety of other key limitations Paolo enumerates tell me that any talk of “zero overhead” is woefully off the mark if you are using FeatureServer.
If you like to program, if you are one of the people who developed the product and if you are willing to price your time at zero, yeah, sure, have at it. But I have to say that the more I dig into the technological details of what has been reported at the beginning of this very long thread, the more I examine the premises and the assurances of everything being hunky-dory and perfect in a way that commercial software cannot contribute, well, the more I have come to realize:
Christopher is a hero, a genius, a true code warrior, for making this thing function, and
This particular process of developing around open source modules has managed to make ESRI’s commercial products seem to be a better value every day!
Jame:
I don’t remotely claim or imply that you are paid to push anything. I’ve met you and I can attest you are a straight shooter and that it is your personal independence of mind that is one of the many things that makes your blog valuable to the GIS community. I apologize if I gave any other impression.
By “ESRI forum” I meant to state the obvious, that your blog does indeed have a predominance of ESRI topics which you initiate. I understand you do so not because of wanting to push anything but because those are the tools you use and you expect are also of interest to readers of your blog. If anything, your recent threads on the many benefits (… dang… almost said it again!
) of open source shows a highly non-ESRI bent.
I was just responding to Vasilis and wishing for commentary from users of commercial software.
To correct a misimpression: it is no one’s job at Manifold to get into DoD or federal government circles because, in general, it is not profitable to pro-actively sell into such circles. We enthusiastically and respectfully, of course, accept orders that come our way from those circles but we don’t prospect there. We don’t even bother responding to RFPs.
The reason why is because Manifold sells on sheer unit volume. When you sell the product for $245 selling 1000 units is only a mere $245,000 at retail, or less than $125,000 in quantity discount. That is not enough money and even less margin, not remotely enough to justify the very high costs of, say, responding to an RFP.
You also have to remember that both DoD and the Feds purchase a relatively small number of units compared to commercial users worldwide. Procurement of 1000 ArcInfo licenses by a federal agency would be a big deal worth tens of millions to ESRI, but that is small beans compared to the volumes sold by us in commercial markets, where 1000 units is not a particularly big deal. It is bigger than average, I grant you, but not something you’d pop a champagne cork over.
There is also a myth that federal procurements drive GIS standards. They do, but only have real effect as regards externalities like interchange formats, which are trivially easy for anyone to add.
Much worse, is what those “big” federal procurements do is distort the product the GIS vendor has to cobble up to fulfill the requirements of the procurement. I’ve read some of those for laughs and I have to tell you, it is not remotely the sort of thing that most commercial GIS users want.
What ends up happening if you chase federal procurements is exactly what happened to Wang with their Wangwriter word processing products. They were so happy to win “big” federal procurements requiring them to do all sorts of utterly pointless customizations that they forgot commercial vendors were cleaning their clocks by paying attention to the needs of commercial customers.
As Wang and so many others have learned, the risk of catering to federal or DoD procurements is that very rapidly you lose the ability to compete in any markets except those. You may become adept at responding to Byzantine procurements involving stratospherically high prices for small numbers of units, but that’s not what competition in commercial markets is all about.
Long term, even with federal markets it is very hard for government vendors to compete against rapidly evolving software based in commercial markets. You risk going the way of Wangwriter and Ada because eventually even the feds will often realize that they can do much better procuring COTS software than dealing with an increasingly legacy, increasingly feeble and increasingly overpriced vendor who has specialized in selling the GIS equivalent of $500 hammers and $2000 toilet seats.
We see that effect already as numerous federal buyers pull out their Impact VISA cards and buy Manifold, just to get the benefit of 64-bit code. They don’t care about the price, but they do care about getting the latest and greatest for their projects.
Christopher:
I sincerely hope as a matter of national security that if you are working for a federal buyer they know enough not to bet the farm on the above two statements.
If the above paragraphs were true, we would not have the seemingly infinite problems with trojans, viruses and malware that afflict just about every organization on the planet from the Pentagon on out. OpenLayers is just JavaScript, and if you can get at the source code you can build especially tricky exploits to sucker the credulous.
I don’t even like discussing this online because sooner or later some 13 year old hacker from Bulgaria is going to find this thread in a search engine, read your second paragraph above and think (in Bulgarian…) “oh, a challenge!” and off he will go to prove in a few minutes through dozens of variations just exactly how much misery can be caused by a malware perversion of OpenLayers interacting with your browser.
Look, I’m not saying there aren’t strategies to fend off malware, just like I’m not saying there aren’t strategies to be reassured about software you do choose to install. I’m just saying that suggesting that alien code executing somewhere else or originating from somewhere else that dynamically loads into your browser cannot be just as devastating a problem as entirely local code is passing the buck.
Just want to make sure the above was not taken wrong as on re-reading it I did not entirely like the tone. Sorry about that.
I strongly agree with Christopher’s comments denigrating WFS. I just meant it is understandable if he has reason to not like it.
WFS is not exactly one of the happiest notions created by OGC. I’d like to avoid it as well and am pushing we do something direct to the Vector layer as an example. But WFS is out there sooo…. we and Christopher and others have to support it. Such is life…
Dimitri:
Can you demonstrate to me a single case where a Javascript-only application running in Firefox has modified the local machine?
Heck, even a case where Firefox has resulted in malware/trojan being installed would be interesting.
I’m not claiming that poorly designed browsers can’t result in software you don’t expect being installed on your computer, but I highly doubt you an cite a single case of the Javascript security model being so poorly implemented that you can use it to remove files from the hard drive against the user’s will.
Dimitri:
You’re now in territory where I am supremely comfortable, so you will be getting an actual response to your comments and concerns this time.
First, some common misconceptions which you seem to be suffering; then, we can move on to seeing why you had problems, and what some of the solutions to them are.
First of all, OpenLayers is a library for building mapping applications. I think that this is one of the most difficult idealogical positions to have people understand: because OpenLayers is relatively easy to write applications in, people assume that OpenLayers is an application. This isn’t true, however, and this shows through in the problems you ran into. The library can be used to do many things, and it has sensible defaults, but any time you are writing an application, you will need to take into account the specific servers you are dealing with. Though it ships with ~100 ‘mini applications’ (the examples) any serious workflow or changes will require some development. “Zero-overhead” is not an accurate description of most tasks in OpenLayers, and that’s by design.
Second of all, OpenLayers is designed to be a tool to integrate disparate datasources. Due to the restrictions placed on the browser security model, we have taken a position of not communicating with the server against the Same Origin Policy — which limits Javascript communication to remote servers — whenever possible. This is, for example, why OpenLayers does not read WMS GetCapabilties documents. (Note that it took quite a few revolutions before we got to the stage we’re at: earlier versions all used GetCapabilities, but this version of OpenLayers has achieved several orders of magnitude wider deployment than previous versions, so I think this can be reasonably considered a success.) It is possible to read GetCapabilities documents (and getting easier in future versions) to extract the information, but in general, that is considered an application level choice, not something that is built into the library. Obviously, this doesn’t work for vectors very well, but for rasters, it allows for the integration of datasources from numerous locations across the web with no server side configuration.
Now, onto the specific issues you brought up:
Not quite accurate: The behavior of OpenLayers with regard to the choice of a default format is expected (though likely ‘poorly documented’) behavior. By default, OpenLayers makes a choice of format based on whether the WMS layer is a base layer or overlay. This works well with the typical servers against which OpenLayers is deployed: It means that base layers are JPEG images (typically low-bandwidth) and overlays are PNGs in non-IE6 browsers, and GIFs in IE6 (which can’t support transparent PNGs). Since OpenLayers is typically deployed in concert with GeoServer or MapServer (which, by default, support all three formats for all layers), this allows for the optimal usage for the most part. Of course, since these are just defaults, you’re supposed to override them if your server requires something different, as you did. However, that’s not a defect within the OpenLayers library, unless you’re saying that overriding the format parameter didn’t work correctly for you: this is by design, and works within the way that OpenLayers is expected to work. It does mean that your application had a minor defect: You didn’t craft your application to take into accuont the server, and so your configuration was using the (insufficient) OpenLayers defaults.
Your next problem was in getting WFS-T to work. Of course, there is a significant problem here as well, at least insofar as your statement that using OpenLayers should provide “instant plug and play”. OpenLayers is designed not for plug-and-play, but for configurability, and this is in no place more true than in feature editing. Feature editing applications are not ‘plug and play’; they are typically simple to deploy, but there is no good OpenLayers editing application in existence which uses WFS-T. (There are, however, a couple of decent ones which don’t use WFS-T, but instead use a RESTful infrastructure to achieve simmilar results.)
However, it sounds like the problem you’re having is likely not in WFS-T, but simply WFS. With that in mind, I am somewhat surprised that you weren’t able to relatively simply get something working, but I’ll admit that the reason that FeatureServer exists is primarily because I know enough to knwo that I can’t get WFS working either. This isn’t a problem with OpenLayers: I simply haven’t read and digested the several-hundred page WFS spec to the extent that I can construct URLs manually which give me the data I’m interested in. Instead, I wrote a geographic feature server which doesn’t depend on several hundred pages of spec: in fact, the way you select features from a datasource is described pretty accurately in just a couple dozen lines of plain text.
Once you get past the fact that loading features doesn’t work, you’ll likely run into a number of other issues with the WFS-T demo: because it was written with relatively limited spec understanding, and no working server to test against, the WFS-T transactions have significant issues which are being resolved now by TOPP’s work into WFS-T support in OpenLayers. Essentially, WFS-T write support is, in the current release, a prototype which should not be expected to work out of the box. It’s significantly better in trunk, and users have reported significant success with building applications around it; I’m thankful that the community has been able to help find these issues and get them fixed (and most of the fixes reported were brought into trunk within just hours of being reported to the development community).
If you’re interested in seeing Manifold used with OpenLayers (which I assume you’re not: I understand that you believe OpenLayers to be foolish software to use, since everyone should be installing Manifold on their desktops
), one thing to consider would be to set up a public demo IMS of Manifold, and then build an example application for the OpenLayers examples directory. I, and presumably others on the OpenLayers development team, would be glad to support you in this (as we have done for other proprietary servers, such as the Cubewerx WMS); if you wish to pursue this route it would be best to set up a server, and report the details to the OpenLayers developer list so that they can try it out. There’s no need for you to spend time developing solutions to all of your problems with OpenLayers (in the same way I doubt you would expect a Manifold user to investigate code solutions to Manifold problems): Our users and developers list are active, with dozens of messages every day, and typically quick to respond to technical questions offering advice and solutinos when presented with a live example of non-working code.
Reading Capabilities documents doesn’t typically offer additional insight to the application one it is correctly built. OpenLayers is designed to allow for developers to build their applications: if they wish to do this by reading capabilities documents, they can do this. If they prefer to simply configure their layers in Javascript, they can do this as well. Reading these documents is unlikely to be related to specific WFS failures.
Ew.
That type of choice would not be the way to go. I’m presuming that Manifold has the ability to serve up collections of features based on a bounding box via a URL in some reasonably sane geographic format. (GML doesn’t count here.) If this is true, then there is no reason to use a Vector layer directly: instead, one can just use something like:
l = new OpenLayers.Layer.GML("Data", "/ims/KMLFrontend?bbox=-179,-89,-1,-1", {format: OpenLayers.Format.KML}); map.addLayer(l); // ... l.setUrl("/ims/KMLFrontend?bbox=0,0,180,90,");Of course, the only thing that OpenLayers cares about for its “WFS” layer is that the layer supports a ?bbox= parameter and doesn’t blow up when it is passed additional parameters (hence the success of FeatureServer as WFS-layer backend): if this is true of a Manifold KML frontend, then you could just use the Layer.WFS with Format.KML as the format class.
Again, if you’re interested in getting some advice, I strongly suggest bringing this discussion the OpenLayers developer list, where smart people can tell you their best ideas on how to get it done… for free, even.
FeatureServer does support storing the data itself, in a couple of ways, but the specific solution described above didn’t use OGR. It used PostGIS, which FeatureServer supports directly, without OGR dependancies.
Agreed. So, since WFS-T is poorly supported in OpenLayers, it would be great to write to Manifold’s RESTful feature API, like the demo that James was discussing did with FeatureServer. A RESTful API, in this case, means you simply serialize a feature, PUT it back to a given URL on the server, and the feature is saved. No need to muck about with trying to describe ‘update’, ‘delete’, etc. operations in XML: instead, the action is embedded directly in the HTTP Request. Any successful vector editing tool I have put together with OpenLayers has used this method of saving features, and I’m very comfortable with it (and this is the behavior that the FeatureServer demo application delivers out of the box).
Is there documentation on any non-WFS-T interface for interacting with Manifold features? Where might I look for that? Perhaps it doesn’t exist yet: if not, I think that if you care about deployment over the web, investigating RESTful access to your data is likely to be significantly more successful for web deployment than suggesting WFS.
I appreciate this; it is a strong statement of support from someone so vehemently convinced that web-based geographic data applications are a thing of the past. I would recommend that in order to best use the resources of your developers, it probably makes sense to begin by contacting the OpenLayers developer list, which is likely able to solve your problems much more quickly, due to their level of experience with the library by comparison, and that contacting first rather than spending time trying to find the solutions may allow for easier integration into the main OpenLayers codebase. I’d also reiterate that such an example problably has as much applicability to the OpenLayers community as it does to the Manifold community, and we would likely be willing to take on the maintainership of a live example if a Public Manifold Demonstration IMS could be made available.
I’m not sure that I’m convinced of that. OpenLayers has not had a stable API change negatively affecting applications in the 2.0 series, to the best of my knowledge, which means that for over 14 months, we’ve kept the same API. An application written with OpenLayers 2.0 should work almost exactly the same as an application written with OpenLayers 2.5 or trunk. So long as you target stable releases, this should continue to be true, likely for another 6 months, after which we will likely continue to support the 2.x branch for some period of time.
If your claim is more that OpenLayers is a developing library, I absolutely agree. The ‘best practices’ for Javascript and web-based GIS are rapidly changing, and the level of functionality we can provide in the browser is growing quickly as well. For example, support for parsing formats of various types really didn’t come around until 2.4, or become complete until 2.5. 2.6 Extends that in many ways, and it’s likely that 2.7 will drastically decrease the level of difficulty in writing support for a new serverside vector feature management protocol. These types of changes are additions, however, and do not negatively impact running code.
If you still aren’t convinced, I’d like to understand the definition of “moving target” you are using, so I can help correct the situation that OpenLayers is creating that makes developing with it as a library difficult.
I agree that there is no such thingas “zero overhead” when integrating multiple tools. James saw it that way in part because OpenLayers and FeatureServer are designed to work well together, and in part because I did the integration. Since I know both parts significantly better than the back of my hand, to him, there was very little integration cost. To be fair, the integration cost of FeatureServer and OpenLayers, even for users who aren’t me, is relativey low: FeatureServer ships with an easily modifiable editing application out of the box, so the cost of integration is at least partially solved, and most development time is spent building the workflow of your viewing and editing application.
Perhaps there is a claim here that I’m missing that Manifold somehow makes deploying a slippy map displaying vector geographic features in a browser more trivial. I know that MapServer does something like this, but MapServer’s “template” mode doesn’t really achieve nearly the level of interactivity that OpenLayers provides, and I think that’s a key difference between the IMS-style services delivered by most existing GIS applications and the results that can be produced by creating an OpenLayers-based application.
Of course, this leads to an obvious conclusion, that any serious IMS style application should be looking to integrate OpenLayers or something similar into their product. GeoServer has already started this process. I’ve offered feedback to ESRI regarding this at any time I’ve been asked, and I would offer the same feedback to Manifold: if you care about integrated IMS serving usable maps, look at how to integrate OpenLayers into your IMS frontend. You’re 100% right that users shouldn’t need to implement this stuff every time: it should be tightly integrated for these type of solutions. That doesn’t mean you’re going to be able to do it in a couple hundred lines of OpenLayers: it means you’re going to need to do some serious work to get it right. An example that shows how to use OpenLayers to use a Manifold WMS isn’t enough: every view of your WMS server should have the ability to be a slippy map, and it should be built into the product, not built by developers after getting the product set up.
A GIS which is designed to avoid the need for system integration efforts should solve the needs of its users. For deploying data over the web, the world has almost entirely moved away from static maps. OpenLayers is a way to offer the ability to deploy data over the web with reasonably low cost, by taking advantage of a product that is commercially usable and connecting it to your existing APIs so that users don’t have to.
Now, onto FeatureServer.
I’ll clarify here that you mean that Paolo claimed that this was the case. FeatureServer does not describe itself this way: the first line of the FeatureServer homepage is: “FeatureServer is an implementation of a RESTful Geographic Feature Service.” It’s not an “OGC WFS”, though it does sort of pretend to be one in some ways.
I don’t think that all three of these are neccesary. Two out of three probably do fine.
More seriously, for developers, FeatureServer has provided an easy way to get started with working with vector feature data instead of rasters. I understand that you’re working in a different world, but for many of us who are web developers primarily, GIS developers secondarily or not at all, FeatureServer provides a way easier leg up on displaying, editing, etc. feature data than Manifold ever will, due to the entirely different nature of approaching the problem. There’s also the fact that most of the developers of these open source products don’t have access to a machine they could use Manifold on, even if they were interested in exploring it; believe it or not, there is a large percentage of web servers in the world which don’t run Windows. 
Your evaluation is reasonably accurate. I appreciate that. I think that your position as a desktop GIS developer (or user? or something? Dunno what your role with Manifold actually is, if any) puts you in a position where the benefits of working with web-centric tools like FeatureServer and OpenLayers are harder to see. I think your position as a Windows-centric developer/user puts you in a position where you’re overlooking a growing portion of the desktop market. (I know that you’ll disagree on that; I’ve read all your arguments, we don’t need to continue it, since we won’t convince each other.) I think that your position as a proprietary software developer (rather than user) is perhaps limiting your viewpoint on the importance of open source software when you don’t have access to the code and development team, or are limited by the commercial interests of a vendor who doesn’t see the need to solve your problem. Given all of these, your evaluation of FeatureServer and OpenLayers is more than fair, it’s downright praise, and I’ll take it as that.
I look forward to further feedback on problems and solutions to integrating OpenLayers with Manifold IMS, and would love to see a public Manifold IMS server with some demo data on it so that I could see how different it is. My requests for WFS servers other than GeoServer and MapServer have, until now, gone almost totally unanswered, and I look forward to seeing that change.
Success!
I promised to report on progress integrating Manifold with OpenLayers using either WMS or WFS. Turns out it is trivial once you know the secret handshake (as is the case with so much software integration). Even as I write this the Manifold user community has already issued a number of demonstration sites.
Adamw, the indefatigable volunteer technical rep on the Manifold user forum, has published the results of our little project to the georeference forum:
To see the attachments for those working with Manifold as well as a link to an example site brought up within minutes by the good folks at mapserving.com, see
http://forum.manifold.net/forum/t60136
Besides the technical info, the link above also has some sociological interest in how we commercial folks talk amongst ourselves about those who have yet to join our ranks.
Seriously, though, the sociology does show three things: First, we respect cool software wherever we find it (note my praise for OpenLayers) and second, there is no hesitation to utilize open source. We enthusiastically grab whatever we like and put it to use. Third, there is an expectation that using Manifold will make that easy to do. All positive.
So, from this day onward anyone who wants to use OpenLayers clientside to front end a Manifold IMS server running either WMS or WFS-T pretty darned near effortlessly.
The above news came in before I saw Christopher’s detailed post, so I will respond to that in a separate post.
Likewise, the availability of such a simple process raises an interesting solution for the proposition James posed to launch this thread. Again, I will put that into a separate post.
We have started the process of filing bug reports for OpenLayers (need to get a login first). However, where does one file bug reports for FeatureServer?
Before I respond to Christopher’s admirably detailed post, a reality check to remind folks what started this thread.
James reported the use of open source tools within an ESRI shop, giving a specific example of what he and Christopher had done to solve a problem.
We’ve been told the customer requirement was to be able to view, analyze and edit features and attributes within an SDE geodatabase both locally and remotely, to publish such information in a web site for naive users and to enable folks to visually select features for download in GIS formats for use with local GIS tools.
I criticized the open source approach using FeatureServer by stating that it was jumping through hoops to achieve less than could be done at much lower cost with a commercial off-the-shelf product. I was told that using the open source tools was the way to exactly meet the requirements of the project without needing workarounds, giving freedom to do what was necessary, with zero overhead to boot.
No one is talking about “zero overhead” anymore, so I won’t beat that horse, but nonetheless all the investigation has given us a basis for comparing workflow between the two different approaches with greater authority and specificity than simply asserting “mine is better.”
Now that we know you can use OpenLayers either with FeatureServer or with commercial servers like Manifold the comparision either way does not depend upon OpenLayers. [The strawman commercial tool I used for my examples was Manifold, but anyone with other tastes can substitute whatever other commercial tool they think is better.]
So let’s make an apples-to-apples workflow comparison between using FeatureServer and using Manifold to implement a feature server to be consumed by OpenLayers.
To create a feature server using FeatureServer:
To get up a feature server using FeatureServer + PostgreSQL you have to install Python and other prerequisites, install FeatureServer, install PostgreSQL, configure FeatureServer and connect it to PostgreSQL, get something that is not WFS (due to the lack of implementation for the GetCapabilities request), configure all that, write the code to make it work using the mind-numbingly complex FeatureServer command structure, etc. For all that work you can’t even use a well-known protocol like WFS and you fail in the requirement to be able to edit features within SDE (you can’t even write features to SDE, let alone have the facilities for multiple people to dynamically edit them). Absent WFS, you end up coding KML as a workaround, and KML is a very low performance, texty, XMLish format like GML. Done.
To create a feature server using Manifold:
Manifold installs via a click-through, consumer-oriented installer dialog. Launch Manifold and link to your SDE geodatabase using a few mouse clicks in the Data Sources dialog. Choose File – Export – Web Page and check the option for WFS. Click OK and and Manifold writes a fully-functional WFS server launch page for you, providing a complete, no-excuses, fully-supported implementation of WFS which is dynamically read-write and which can write to your SDE geodatabase. Done.
Adding support for OpenLayers:
This seems to be a wash.
In either of the above cases you have to add some lightweight support for using OpenLayers. The Manifold case is easy, by cribbing the 50 or so lines from the recently published example.
I don’t know how many lines it takes with FeatureServer’s need to use a KML workaround, but I’m willing to stipulate that the effort would be similar.
Exposing content to the Web:
If you want to expose the contents of your geodatabase through a wide variety of web sites, no problem with Manifold: use that File – Export – Web Page command to build simple, but effective, web sites or use your programming skills to create whatever elaborate thing you prefer. You don’t need to use OpenLayers, but you could if you preferred that.
No doubt FeatureServer can be rigged up with OpenLayers to also expose content to the web. It appears to require more programming but that’s a quibble and I don’t think it matters to my argument if FeatureServer has slightly more, slightly less or about the same level of programming required.
The bottom line:
The FeatureServer approach requires acquisition, installation and initial and continuing integration between several packages (Python, PostgreSQL/PostGIS, etc.). It requires programming to serve features by supporting KML interchange, for lack of WFS. It fails the requirement to connect to SDE.
The Manifold approach requires acquisition and installation of one package. There is no integration because that has been done for you and will continue to be done for you. The package will work exactly how documentation says it does, with maintenance and bug fixes performed by the vendor. It requires no programming whatsoever to serve features (WFS) out of the box. It connects to SDE out of the box.
I don’t think that for real world GIS work as most people do it there is any contest. The open source approach reported does not fulfill a key requirement (the data is in the SDE geodatabase after all) and it costs a fortune in time and integration tinkering. The commercial approach costs a mere $395 and can be done start to finish in a few minutes with virtually no programming except that imposed by a requirement to use OpenLayers.
I therefore stand by my original criticism that it would have been more effective from a business perspective to not have jumped through hoops to attempt this with open source tools but rather to consider a case where this particular ESRI shop could have had its ESRI work enhanced through the use of some supplemental commercial tools.
There’s nothing radical in that proposition, since ESRI shops around the world on a routine basis utilize commercial tools from other GIS vendors as part of their toolkit. Commercial tools will often provide out-of-the-box interoperability with ESRI technologies, such as SDE, in a way that open source tools do not.
A final note: The above comparison is if you must use OpenLayers. An even simpler way to solve the requirements of the customer would be to simply install Manifold on the customer workstations that need to edit, to completely remove the need to use OpenLayers. In that case you are done in one step without even the need to spend any programming time configuring OpenLayers. Simply install Manifold on the client machines and then they can connect directly to the SDE geodatabase.
I realize Christopher has amended the customer’s requirements to outlaw the use of anything other than what he advocates, but I say that is a red herring, discussed in a different post.
Dimitri:
You’re acting as if all of this development is happening in a vacuum. Now, I’m not in a position to claim that your statements about ‘typical GIS setups’ are or aren’t true: to be perfectly honest, I’ve never done any GIS development in my life, so I don’t know what the heck a ‘typical GIS’ shop looks like.
However, I work with a lot of people who are already managing data in PostGIS, and that’s what they want to make available. Or they’re already managing data via OGR, and rendering it via MapServer, and that’s what they want to make available.
If you want to narrowly focus on one specific use case, and ignore all others, I’ll admit that it’s possible that Manifold would be an ideal solution. This is equally true for all solutions.
The decision to use FeatureServer in this case was specifically due to a lack of a need for ArcSDE support. Sure, it might be handy at some point in the future to have support for ArcSDE, but (at least as far as I’m aware) the client wasn’t asking for it in this case.
For users who are regularly working with Open Source geospatial tools, setting up Python and PostGIS is trivial: most users who would choose to use FeatureServer already have it configured. Same is true of Python. If you don’t have this situation, there’s no reason that you wouldn’t just use GeoServer — since GeoServer, like Manifold, supports writing directly to ArcSDE, and supports WFS-T.
So, why use FeatureServer over these other tools? There’s a pretty simple reason: The developer you’re working with is most familiar with it. If you’re looking for a solution to deploy on a Linux webserver (because that’s what your client has) and you have PostGIS and Python installed already (because installing them is relatively easy, and you may need them for something else anyway), there’s no strong reason not to deploy FeatureServer: For users for whom this is the case, it takes just one command to deploy FeatureServer. Heck, even if you don’t have PostGIS installed, depending on your use case, it doesn’t really matter: FeatureServer has a built in method that works fine on any platform that will support up to 20,000 features without any non-Python dependancies. No database required.
If you’re more comfortable with Python than Java, and you’re more comfortable with Linux than Windows, FeatureServer is likely the easiest choice to get a working feature-editing setup completed quickly. If you really don’t believe that, I’ll make a screencast of the full process needed to get FeatureServer running on Windows to prove it.
You’re comparing apples and oranges. Comparing Manifold to FeatureServer is like comparing ArcMap to OGR. FeatureServer is a tool in a toolbox that allows you to build applications quickly. Manifold is an end-user application for working with GIS data.
You say:
Now, let me take that paragraph and rework it for GeoServer, in my experience:
Amazingly, these are more similar than the Manifold/FeatureServer comparison because the two tools are more similar.
You say:
You don’t have to use KML to use FeatureServer. It’s clear that you’ve put no effort into learning what FeatureServer is or how it works. Using the exact same line of code that you used to connect to Manifold’s WFS, you can connect to FeatureServer’s WFS:
wfs = new OpenLayers.Layer.WFS("WFS", "featureserver.cgi/scribble?format=WFS", {maxfeatures: "50"}, {extractAttributes: true, displayInLayerSwitcher: false});How is that difficult? The fact that FeatureServer “doesn’t implement WFS” doesn’t bother OpenLayers in the slightest, since OpenLayers doesn’t either. The fact that the FeatureServer demo, linked form the FeatureServer homepage — is longer in pure ‘lines of code’ is simply because it has significantly more functionality, like attribute editing, links to different formats of features, and the ability to interrogate features for information. The demo is, of course, developer oriented: FeatureServer is a developer-oriented tool, so the demo is designed to accomodate for its audience.
Sure. And “Real World GIS work” would never be done using Google MyMaps. But Google MyMaps gets a lot of daily use: on a day to day basis, I would say that there are significantly more people in Cambridge using MyMaps than there are using any ESRI tool.
I haven’t amended this customer’s request: I’ve pointed out that although I helped James out on this one, this isn’t my job. I highly doubt that anyone reading this has any clue what I do on a day to day basis, so I was taking advantage of your experience in understanding whether Manifold helps me solve any problems that I need to solve. It doesn’t. This is equally true for a number of other people I work with as well. FeatureServer and OpenLayers, on the other hand, do help me do my day job, and do it reasonably well, and helping to understand which problems they do solve is important (at least, from my point of view).
Sure, your posts started with “Manifold could do this particular task better.” But they grew into “Open Source is useless” and “FeatureServer and OpenLayers don’t solve any problems that Manifold doesn’t solve better.” If you really still see me as arguing the first point with you, then you should re-read my recent comments. But you’re still making statements that make it clear that you hold the latter two positions. You can’t see open source code libraries as tools in a more widely applicable toolset, used to solve a wide variety of problems. And so long as that’s the case, we’re going to continue to disagree.
Christopher, I think we are interleaving long posts but what is below responds to your prior detailed post:
I am grateful you would take the time to write in detail. I know from my own experience writing in the detail sophisticated concepts deserve is not easy, and I deeply appreciate that you respect this community and your readers enough to be willing to invest your time in a discussion that is to everyone’s benefit.
In what follows for the sake of time I have not edited deeply, so up front let me state that I wish to be respectful and that any “tone” that comes across other than respect is simply an unedited attempt at the occasional colorful phrase so that folks might not completely fall asleep. It is not intended as disrespect.
First, a clarification to avoid word games:
I previously stated the WMS standard specifies that the client should ask the server which formats it supports. Strictly speaking, that is implied but it is not explicitly required as part of the standard. I call this a word game because the standard does not specify any other way to ask a WMS server which formats it supports except GetCapabilities so the only possible inference is that this is what should be used.
It is true that a client communicating with a particular server in a “hard wired” integration setup can know these formats beforehand; however, the point of standards is to address general cases where the client might not know in advance what server is used so it can be hard-wired to use that server.
It should also be obvious to all that having to specify a format manually to the client in hard-wired form prevents a server from changing formats, say, during an upgrade. Suppose, for example, version 1 only supports BMP, but version 2 supports JPEG and PNG, and version 3 which is currently in development will support a future format that is as yet unknown – Given automatic detection of the format using the GetCapabilities as implied by the WMS standard, the client does not care, as long as the format output by the server is supported by the client. And of course, there are reasonable automatic ways of choosing one of the formats supported by the server when it supports multiple formats. There is no need to force that choice upon the user by displaying all possible choices and having the user pick one.
I say it is just a word game to play on the difference between explicitly stating GetCapabilities should be used or implying that it should be used because it is the only such facility offered by WMS.
If someone disagrees with that, I am willing for the sake of argument to stipulate that it is not officially part of the standard and to amend my criticism of any would-be “WMS client” to say that failing to utilize GetCapabilities shows both total disregard for the spirit of the WMS standard as well as an absurd lack of common sense.
Let’s touch on specific points advanced by Christopher:
The policy here is that browsers prevent a web page coming from domain A from manipulating documents and issuing XMLHttp requests to pages in domain B (or on domain A but on a different port, or on domain A but on HTTPS instead of HTTP, etc). The WMS spec says that the GetCapabilities request should return URLs for other requests, such as GetMap, and that these URLs can be different from the URL used to service the GetCapabilities request. This makes sense and is an important feature, because it can be used for improving performance via load-balancing and for many other useful things. In addition to that, the GetCapabilities request is supposed return the formats supported by the server and other useful data. The client should use the returned data to make further requests.
It could be I misunderstood Christopher’s intent, but What he appears to be trying to say is that since OpenLayers works in a browser, all requests it will issue are subject to the “same origin” policy enforced by that browser, and thus it does not make sense to do the GetCapabilities request because if the URLs returned by that request differ from the page URL, the code in OpenLayers wouldn’t be able to use them. I grant you there is some sense to that, but really this just means that it is not a good idea to do WMS in the browser (and possibly also that WMS is a bad protocol). The bottom line is that if you want to show images obtained from a WMS server, OpenLayers is a bad choice since it has no chance to work right unless the server satisfies certain conditions that are not in the standard, and for which there is no reason not to be in the standard (the load-balancing thing).
In any event, why throw the baby out with the bath water and ignore not only the URLs but also the formats returned by the server? Isn’t flexibility supposed to be a strong point of OpenLayers? Why throw away the autodetection code instead of using it by default and only have the format overridden if the user specifically indicates the desire to do so? This would not only make it easier to wire OpenLayers to WMS servers for newbies.
Note that you can say all you want about the developers of OpenLayers being in agreement that the choice of the format belongs to the user, but this secret covenent is highly opposed to the open interoperability ideal for which WMS was introduced. We can see that in actual practise it is making it more difficult to wire OpenLayers to WMS servers, and it doesn’t have to be that way. It would also allow a web application using OpenLayers to work if a server updates the formats it supports, and there would be other benefits as well.
End users want plug-and-play. This is the “awk and sed” vs “Word” thing all over again. There is a lot to be said about integration, as evidenced by an example of publishing a WFS server using FeatureServer with any of various storage back-ends (multiple packages, each of which is supposed to do its thing right, yet with many of these packages not living up to what they are supposed to do, a fantasy in practise) vs Manifold (one package, no code, integration issues are minimized, outcompetes FeatureServer on both front-end [WFS] and back-end [SDE]).
Look, even with the “open source” community the whole idea of WMS is to encourage interoperability. Unless you are going to be squishy about this and say that “interoperability” means not reducing custom programming through plug and play, but rather just means robbing Peter to pay Paul by avoiding programming in one spot yet having to do more custom programming in another, well, then you have really devalued what people are trying to accomplish with standards like WMS.
That is a painful thing to read. I winced when I read that because I admire Christopher and it is not easy to see someone you admire, well, giving up.
How do we reconcile this comment with this statement on openlayers.org:
What is painful about this is that here we have a serious proponent of open source software on the one hand being overwhelmingly in favor of (open) standards and wanting them to be supported to the letter, but we also have the unexpected situation of a commercial product that supports “the standards” to the letter and at the same time an open source library that does not support them – yet that same proponent, who is one of the persons behind the library, is stating that this lack of support is due to the lack of time to read and understand the spec. Ouch.
The front page on openlayers.org also states that:
How exactly is that going to happen without the standards? How exactly is OpenLayers going to interoperate with products other than FeatureServer?
Further down we also see:
Is it that OpenLayers is meant to operate with a specific commercially usable product, but not with any product supporting a certain (open) standard? That would be quite a revelation.
Look, I take all the above with a certain grain of salt because it proves the truth of two of the things I wrote in my famous letter to the editor about open source: the first is that it is hard to muster the density of effort required to pull certain complex things off via open source development. The second is that commerical companies at times use that effect to hijack open source work to their benefit.
Christopher, you are right: you don’t have time to read, digest and then perfectly implement servers and clients that 100% fulfill the WFS spec. That has been a classic weakness of the noncommercial movement. At the cost of millions of dollars and throwing a large, well-funded, full-time development team that works night and day six days a week Manifold can afford to do so. It costs a lot of money to concentrate enough talent in time and space to even understand those specs let alone implement them. That has been a classic strength of well-run commercial efforts.
But another weakness of noncommercial work is that folks are often too quick to pretend away or redefine whatever failure there is to fulfill what is stated about your products. You get to say “oh, yeah, it doesn’t do what I said when I wrote…” in case like:
It does not work to say “But I had my reasons for doing so and how I did it was easier for me than reading that danged long spec and doing what it said. Like, what, anyway? …it’s free software, don’t you know?”
As a commercial company, we don’t get to do that. When we sell software that we say does WFS, it darned well better do that because we took the customer’s money. If it doesn’t that’s a bug and we fix it. No way does Manifold or any other commercial vendor get off the hook by saying the spec was inconveniently long to read so it doesn’t matter that the software doesn’t do what we said it does. (!)
The commercial companies like ESRI who play puppeteer with OGC understand your lack of time perfectly, and that is one reason why they strive for extremely dense specs that no one but them can understand and implement. Their strategy is to bribe the government and other institutional buyers into requiring a given spec as a way of assuring that only their products can be sourced. It is a barrier to entry to both smaller operations and also even to larger operations which have more modern technology than the living fossil junk cast into amber by OGC.
I hate it when they do that and I am strongly on your side regarding your comments about WFS. But we cannot play both sides of the matter: if we are going to use support for standards to encourage uptake of our programs we have to support those standards, even if we think they are dumb.
We’re completely neutral on that. I mean, REST is more or less just using HTTP verbs other than GET and POST to manipulate data stored by a web server. That’s an old concept with not much controversy about it so I don’t see anything to get heated up about over it one way or the other. Manifold does not automatically support REST with template code, but it is easy to add that support to any Manifold web application since Manifold does not place any limits onto what can and can not be done in the context of a web page providing such services. Sure, there are limits on features related to the interactive user interface but no problem coding up RESTful services if that is what you want.
If the user community ever wanted Manifold to provide a checkbox for REST to automate this the way we do WFS and WMS and image servers and other things, well heck sure we’d do it in a heartbeat. Traditionally, we want to listen to what users want because there are very many ideas out there that never get traction so there is no point investing resources into something our users do not want when there are plenty of items on the wishlist they would prefer we implement first.
That depends on what users think, and if they want it, sure, we’d consider doing that for the next release. It’s easy to do and if it saves our users one small integration task should they want to use OpenLayers, we’d be happy to do that. We’d like to see how this plays out because (see my other posting) there are already very viable alternatives for solving all problems OpenLayers is aiming to solve. But if our users wanted us to, say, provide a web site template which implemented features specific to OpenLayers, time permitting you bet. No promises, obviously, as ultimately this is up to the same prioritization process that’s driven by the expressed will of the Manifold user community.
Could you elaborate? I suspect we already do what you want but without understanding the comment better I can’t say for sure.
and
There is dense detail in the programming reference in the manual but better still to have the real thing at hand to try out when that is useful. If you could contact me offline at dar@manifold.net I’d be happy to arrange some licenses for you at no charge. Manifold has always provided no-charge editions to journalists and to development partners at infrastructure companies like Oracle, Microsoft, NVIDIA, the printer companies, etc. We would be honored if that would help integration. I think you’ll be shocked at the degree of utterly arrogant Microsoft-centrism, but well, part of integration is getting to know these little cultural differences as well.
Thanks for taking the time to write in detail. I still don’t agree with you on a lot of things but as interesting as it is to discuss different approaches I think we both agree on OpenLayers being a cool thing. I have no hesitation praising things like OpenLayers. I’ve criticized how OpenLayers does the WMS and WFS things but I think that on the balance those matters are nits and that it is a terrific, rather well implemented idea.
Dimitri:
Thanks for your response. It’s good to see that you do seem to take OpenLayers seriously, and I appreciate the depth of your comments here.
You state:
I accept this. However, OpenLayers is not, primarily, a “WMS Client”, nor is it primarily a reference implementation for geographic data standards. Instead, it is a tool which can be used to generate a web-based view onto geographic data. As such, the development of OpenLayers concentrates on doing that one thing, and doing it well.
Other users have used the OpenLayers library to build WMS Clients: these clients allow you to enter a WMS ‘Endpoint’, select a layer, select your preferred format from those returned by the server, and serialize these settings out into reusable OpenLayers application code. This is an example of an application that can be written using the OpenLayers library. Alternatively, an application can use the available code to parse WMS capabilities, and use these to configure their application. For some users, this makes the most sense, but not for all.
Regarding the lack of ability of Javascript applications to load remote XML data, and OpenLayers specific decision to take this into account in its development, you said:
I disagree. It is possible to find solutions to these problems: in fact, Manifold would have no such problems, making the use of WMS GetCapabilities a reasonable choice for building an OpenLayers-based Manifold WMS client. However, since it is unlikely that any application will be doing things like changing supported formats on the fly, or adding additional layers that an application will want to pick up automatically without any developer interaction, it makes the most sense to develop an application with the choices made by a human in the loop.
Huh? I don’t know what load-balancing thing you’re talking about, so I’m not sure how to respond to it, but certainly humans who create OpenLayers applications can easily make the decisions about how they want to use WMS, and instruct OpenLayers in the rules it needs to follow. Certainly, that’s what I do, and I don’t have any serious problems with it.
Huh? That’s exactly what OpenLayers does. Did I mislead you somehow?
Again, you’re saying “user”, I’m saying “application developer”. An application developer is required if you want to create an OpenLayers based solution, because relatively few applications exist which are built on top of OpenLayers yet. Application developers have to learn how to use their library, and that includes specifying how to talk to the server.
Perhaps your claim is “OpenLayers should be developed as a WMS client, not a geographic data viewing library”. In fact, many of the claims in this section seem to be claiming exactly that. That’s a difference of opinion, but I’m happy with the choices OpenLayers has made. Similarly, the following follows into the same vein:
End users are not a target audience of OpenLayers. Web application developers are. When I write an application based on OpenLayers, it’s definitely not going to look like OpenLayers does. It would likely look more like TileCache: TileCache has very explicit instructions on how to deploy it, how to configure it, what various configuration parameters mean, etc. This is significantly closer to an application, though it’s a very limited application (since it’s a server designed to be used by remote clients).
How does the Open Source library not support the parts of the specification it implements? You got OpenLayers set up talking to Manifold in what appears to me to be a couple hours. Given that OpenLayers is developed for an entirely different toolchain, I don’t see this as a problem, and in fact, it seems to me like OpenLayers must do relatively well at implementing the standards in question, otherwise you wouldn’t have been able to get it working with Manifold without some kind of modification to the library.
Well, it looks to me like OpenLayers integrated with Manifold fine. It works with MapServer, Cubewerx, GeoServer, etc. It works with OpenStreetMap tiles, it works with Google Maps, it works with Virtual Earth maps. The “How” is an application developer who cares to make it happen, because OpenLayers is not an application, but a library, and all libraries need applications to be useful.
Huh? How does OpenLayers fail to work with Manifold? It needs a bit of documentation on how to use, and therefore it is in violation of the spec?
And we’ve done the same thing, by creating a product that has successfully, with no additional library development, integrated with Manifold’s WFS and WMS servers for read and write of geographic data. Why is that? Because I didn’t write the WFS support for OpenLayers — someone else did. In fact, it was a commercial organization who wrote the original support, and a collection of commercial and non-commercial contributors who succeeded in pushing it farther than it was pushed by the commercial user. How is this an argument against open source, exactly?
The fact that I personally don’t use the WFS spec, preferring AtomPub as a protocol for web-based feature management, isn’t a statement on OpenLayers — it’s a statement on me.
This statement is true. It may not tell the whole truth in your opinion, but the method by which OpenLayers loads remote data is via industry standard request mechanisms. There is no question in my mind, regardless of what you say, that OpenLayers is holding up its part of the bargain as a library.
Are you serious? Commercial providers get away with poorly or partially implementing specifications all the time — if they bother to implement them at all. Claims and press releases are made every day about commercial software implementing standards that it doesn’t. There’s no question in my mind that this statement — regardless of the truth as regards to Manifold, which I can’t speak to — is patently false in a more general sense.
Sure. Sorry: I was offering a personal suggestion as a web developer, a community which appears to be under-represented in your current set of Manifold users. As a web developer, AtomPub is a much more useful API than WFS-T.
Sure. Again, the idea of using OpenLayers in an IMS frontend is a personal recommendation. Users want slippy maps, and if you care about supporting web-based usage of Manifold, I’d recommend adding slippy map support. The next comment (#6) is the same topic: To really get people excited about your IMS support, it should be fancy out of the box: it shouldn’t need an application developer to make it fancy like Adam or your other developers did last night.
Although I appreciate the offer, you haven’t picked up on the key part of something I’ve been saying: unless you can arrange for some Windows licenses for me for free of charge as well, I’m SoL on trying out Manifold, as I don’t have access to any Windows machines in my work. However, I would be interested in helping offer advice on what a sample IMS Server which Windows- (or Manifold-) challenged developers could work against might look like. FeatureServer does this, as does TileCache, and both services have proved invaluable for client developers to understand the services available from these products without needing to install them themselves. It’s also how we did a majority of the development of the WFS-T support, using servers provided by TOPP (GeoServer) and OpenMNND (MapServer): if one had been provided, we would also have developed and actively supported Manifold. Luckily, we got the spec ‘right’ enough that we didn’t need to bother, but setting up servers with documentation for web developers is a great way to ensure that web applications support your server.
OpenLayers implements WMS and WFS enough to allow an application developer do what they need, and I think that’s appropriate for what it is. I appreciate your forthright positions, even if we disagree, and I’m glad that you see that there is at least some benefit to be gained via a library like OpenLayers.
Christopher,
Wow. That is genuinely a fork in the road. Given that most people use Windows it is quite rare to encounter someone who has no access to Windows at all.
Well (sigh) I wish you luck with that. I appreciate your offer of advice.
Dimitry.. welcome to the Real World .. not everyone uses Windows.
And Chris is proving all your points to be false because he is doing everything you are doing… and NOT using Manifold.
And I do have Windows.. but I don’t use commercial software..
But this has all been an interesting read.
Christopher did not prove any of Dimitri’s points to be false. Please read the thread.
I am glad the discussion has morphed into a technical one, at least in part. As regards arguments, I am on Dimitri’s side, but I don’t want to be dragged into a large battle and will instead try to answer some technical questions asked by Christopher.
Here goes:
“C: Huh? I don’t know what load-balancing thing you’re talking about, so I’m not sure how to respond to it, but certainly humans who create OpenLayers applications can easily make the decisions about how they want to use WMS, and instruct OpenLayers in the rules it needs to follow.”
The load-balancing thing is a dedicated entry server which responds to the GetCapabilities request and a set of servers which respond to GetMap requests, which the entry server attempts to load equally. Eg, “caps.mywms.com” and a set of “mapX.mywms.com” for X between, say, 1 and 4. The client starts talking to “caps.mywms.com” which refers it – via the capabilities document – to one of the other servers. OpenLayers would not support this configuration simply because it is subject to the “same origin” policy you talked about. As it stands now, OpenLayers does not support this configuration even if all of the servers responding to GetMap requests would pretend to share the same domain (no conflict with the “same origin” policy), since OpenLayers does not request the capabilities document. WMS clients that do request the capabilities document (yes, this includes Manifold) support the above configuration automatically.
“D: Why throw away the autodetection code instead of using it by default and only have the format overridden if the user specifically indicates the desire to do so? — C: Huh? That’s exactly what OpenLayers does.”
Dimitri meant issuing a GetCapabilities request and choosing a format from the list of formats that the server actually supports. OpenLayers does not do this.
Okay, this clarifies the point: I see what he meant now. I was never intending to say that the autodetection code should go away, so I’m not sure if this was responding to something I said unintentionally or what, but at least I understand now.
Chad:
You mean like the dynamic connection to SDE that Manifold provides? … Just to be sure, I checked again and I don’t see that in the FeatureServer solution.
That’s the most obvious example from what little we know about the actual application but I’m sure if we were to examine the actual application we could find, oh, perhaps a few hundred other examples.
I find it interesting that an open, creative solution to a real-world problem would spark so much discussion.
Ask yourselves. Is there “one best way” … to solve a problem, or are there many “good enough” solutions that meets the needs of the customer? Pick one, and be happy. In recommending solutions, we should always be laser-focused on the question — did I meet the customer’s needs?
If I have, and if I can leverage what already exists, then I have done well. If I have provided a solution that will continue to meet their needs into the future without extensive, expensive maintenance, I have done better. If my solution can flexibly evolve as requirements change, I have done awesome!
Do open technologies give us more / better tools to use?
Do they allow us to integrate with existing patterns and workflows? (sometimes)
Do they result in lower operating and maintenance costs, without sacrificing support entirely? (you get what you pay for … can you live with that?)
Do they allow us to flexibly evolve the solution?
I think so, but that’s just me.
Am I wrong in saying that FeatureServer wouldn’t add much more flexibility for a group with access to ArcGIS Server 9.3 with its new features (RESTful interaction, Javascript APIs, KML output)?