That was an email I got last night from a reader. They like all of us were amazed at what she was seeing. The solution is not that hard to imagine though. Most of the problem we run into is processing power and bandwidth. None of us can afford the server farms and the racks and racks of servers that Google can deploy. But that isn’t a problem. If I were ESRI, I’d be looking at allowing their customers/clients to use Amazon Amazon Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2) for hosting and processing. The cost of such services is amazingly low and you can take advantage of Amazon’s infrastructure to offload some of the processing on it. Sure it won’t be “Google Fast”, but it will get you as close as you can get without being at Google.
About
James Fee - jfee@weogeo.com
James works for WeoGeo helping people organize, share and monetize their geo-content.
Connect
Subscribe
25 Comments
James, I love your insight. This all seems very possible if ESRI would just listen to you. For the life of me I don’t know why they don’t hire you. ArcGIS with AWS just makes me smile.
A2E already uses S3 for hosting so I suspect we’ll see quite a movement into this arena in the next year.
Especially with Google pulling this kind of stuff off. You need some horses in the shed to do what they’ve done and all the readers of this blog could pool their servers together and not even get a small part of the computing power of Google.
These scalable services have been around before, but Amazon has the price point right now we can work with.
I agree james…but how long do you think it will take for ESRI to implement this assuming they want to.
Who knows, I doubt ESRI has even thought about doing something link this.
I’d say “never going to happen” is probably the most likely scenario.
Now that doesn’t mean that other GIS applications might not go this way. You just have to keep your eyes out.
well, the reason no one can compete with google in terms of computing power is because no one wants to think like google. Their stubbornness will be their downfall. Google is location data centers in places that are very cheap to operate…like Pryor Oklahoma. Energy is cheap, wages are cheap….in short everything is cheap. ESRI thinking is you have to locate in trendy places and as we all know trendy = $$$$$. For instance, could have their user conference in a less trendy place than San Diego….but that wouldn’t be hip or cool….
As long as companies aren’t willing to think the way google thinks, google will win this battle.
Here’s how to compete with Google. Turn to a socialistic model. It’s not fair Google has all that computing power and others don’t. Petition the govt. to restrict Googles computing power or even share or give computing power to their competition.
Would that work?
Or how about this. Convince Al-Qaeda types or ELF to bomb Google’s server farms. I’m sure it wouldn’t be hard to convince them. ELF wouldn’t like the development, more buildings, or even the huge use of resources to power those farms. And for the other guys, I’m sure I could point to something Google does which they wouldn’t like.
Would that work?
It’s my Friday outside of the box.
KoS
Otherwise….the deal Amazon has seems like a win-win for the small guys. Helps lower the barrier for others to step into the mix.
KoS
Q: How can I compete against google?
A: The same way you stop earthquakes, hold back hurricanes, or control tornadoes – you don’t.
I certainly don’t have an MBA but I wouldn’t think that ‘anti-Google’ is a good business model.
This is the price ESRI pays for not co-opting a proactive web delivery solution in the 1999-2002 time period. With no clear-cut market dominance from ESRI, Google saw opportunity and methodically revolutionized how GIS is delivered to the masses.
Think about it – while ESRI was finishing the touches on ArcIMS 4.x and grudgingly moving towards open-standards in IMS 9x – KeyHole and then Google brought GIS to the masses in a one-two-three type process that literally kicked down the doors of what had been a nearly closed-door society of distributed GIS solutions for those with big budgets.
There can be no denying that Google revolutionized how GIS is used around the world for decades to come (an extreme over-simplification ?). And I would doubt that they are done.
If one had to ascribe one or two reasons why ESRI let this happen – I would suggest that ESRI was focused on the desktop while Google never had any desktop baggage to maintain. The only time Google goes to your desktop is to add value to the Google brand.
I’m not anti-ESRI, I’ve just been extremely disappointed at their web-based solutions for a number of years.
ESRI obviously couldn’t ignore the desktop – that was their legacy. Dependency on the desktop mandated a focus on the server – but the distribution model outside of the organizational desktop/server was not properly emphasized. I would stridently argue that high accuracy data creation and editing SHOULD be done at the ESRI desktop and with ESRI enterprise storage and delivery within the enterprise. However – mass delivery of my ESRI-maintained data better have an interface with Google Earth and Google Maps or else.
Why fight Google? It’s a loosing proposition.
Is Amazon web services really going to compete with Google? It’s an alternative – probably a very good alternative…but I question the perspective of “instead of Google” vs “in addition to Google”.
I’ve seen mention of IBM considering ESRI as a possible acquisition – Perhaps a better question is when will Google buy ESRI ?
James:
Per-seat (not to mention per-processor) licensing is too lucrative to give up until absolutely necessary. And getting fat on selling features the majority rarely use has worked out well enough. And if you need to offer specific tools to help customers “manage” their licenses–you’re probably a ripe target for a competitor’s software-as-service model.
When GIS people express frustration about their job, all too often what they’re really saying is “while I love the GIS part, I hate sys admin, I hate managing services, I hate diving into OOP for lightweight customization, and I hate combing through log files to figure out why things aren’t working. And I really, really hate when the suggested ‘fixes’ include fiddling with registry settings or obscure IIS parameters–more complexity and techie minutiae that shouldn’t be my problem.”
Which was the genius part of the Arc2Earth approach using S3–Internet mapping with no server management involved.
If I were a betting man, I’d put my money on someone putting together a PostGIS/MapServer/GRASS stack on EC2 and selling tile making, interpolations, routing, etc. based on usage.
And only when a big customer stops the annual licensing fees because this “faster/better/cheaper” approach actually works, will the entrenched give serious attention to an alternative model.
Brian
So when are you going to release that Brian?
OK, this is a brilliant idea. ESRI should be offering this kind of service or allowing their business parters to do so.
I think what James was getting at (feel free to correct me) is that we (ESRI users) need to keep our customers from defecting to other services. It isn’t so much that we care if we are using ArcGIS or QGIS, but that we want to be those folks implementing it. One call to Google and we are out of a job. My little company can’t afford to make the investment in programmers to make such a jump so we need our partners (ESRI, AutoCad, Oracle, Microsoft) to do that for us.
Keep pushing James!
Kipter Uh said: “we need to keep our customers from defecting to other services.”
It seems like the age old trap of letting the software drive the solution.
If your customer asks “What about Google Earth” or “Google Maps” – I hope your answer isn’t “…because it’s not ESRI or Autodesk”.
If the customers like “other services” – you better find a way to insert yourself in to that supply chain or you will lose customers.
[Insert random quote ref: "change is the only constant" or "failure to adapt = death" here]
To be fair Paul, it sounds like he is looking for his partners to help him find that solution.
James:
This is the exact reason that I have been waiting so patiently for my acceptance into the EC2 beta. I was rewarded on Monday. I am in….
However ESRI still has some barriers. We as users/business could publish a public AMI of ArcGIS server if ESRI would publish a time bombed AGS install, much like Microsoft does for all its virtual machines. EC2 does still have some issues such as dynamic ip’s, and a lack of persistant storage at the EC2 instance level. You have to use S3 and some cool master/slave/warm backup setups of your database.
That brings me to another point, ESRI has defiantely followed the Gov. crowd into the .NET world. You can’t work a model like EC2 if you still have to license all those OS’s. I guess all those C# guys can give up visual studio, grab Eclipse or NetBeans and go for it with Java (it is so close already). Perhaps Mono will get some love..
I don’t think the .Net crowd has got the community to do it though. And as long as ESRI is putting most of its effort in that camp it doesn’t look good.
My 2 cents
James wrote:
“ESRI obviously couldn’t ignore the desktop – that was their legacy. Dependency on the desktop mandated a focus on the server – but the distribution model outside of the organizational desktop/server was not properly emphasized. I would stridently argue that high accuracy data creation and editing SHOULD be done at the ESRI desktop and with ESRI enterprise storage and delivery within the enterprise. However – mass delivery of my ESRI-maintained data better have an interface with Google Earth and Google Maps or else.”
This is so true
This is a new product for ESRI Desktops: High accuracy 3D stereoscopic data capture, something like Leica StereoAnalyst or the SOCET SET Extension but affordable nad end user oriented: Purview -> http://www.mypurview.com ESRI Canada is already distributing it
The postgis/grass stack on EC2 would certainly be a fantastic tool, one worth pursuing. The way you can create new servers instantly is very compelling. Being able to create an open source stack in minutes would be in stark contrast to the horror of Arcgis server installs. However, it is just as important to simplify the installation on other platforms, as Postgis is still very difficult to install.
Very intriguing paradigm shift indeed. It’s easy to see the benefit on the application side of the house but what of the rdms? I would love to get out of the business of system architect/network admin/everything else non-geospatial required to implement and support an enterprise GIS. However, how does one migrate terabytes worth of data to the S3 site and manage updates accordingly? Regardless I love the idea of hot and cold running system resources. Hmmm, come to think of it the amazon web services business model is strikingly similar to that of utility companies. Wouldn’t it be interesting if this model grows and becomes the norm and politics enter the scene; it’s easy to imagine the talk of nationalizing such resources (enter Hugo Chavez) or creating public internet utilities. It’s a brave new world.
As for competing with Google… why fight? I find it curious how many GIS professionals view the new commercial GIS apps (Google Earth, Virtual Earth, et et) as a threat rather than an asset. Our network staff doesn’t “compete” with dell, ibm, sun, microsoft, linux, et et et. They assist business experts in leveraging their resources toward specific requirements rather than try to keep up with the vendor’s product lines and services. If Google Earth meets someone’s specified requirements more effectively and efficiently than ArcIMS; great. What’s important is their problem is solved. This may be an over simplification but I am relieved to pass the torch of supporting ubiquitous and generalized web mapping applications to the googles, yahoos, mapquests, and microsofts of the world so I can concentrate on the more complex and specialized solution requirements that the suite of commercial applications can’t address. The everyday web map has been commoditized… hallelujah.
“As for competing with Google… why fight? I find it curious how many GIS professionals view the new commercial GIS apps (Google Earth, Virtual Earth, et et) as a threat rather than an asset.”
Well put. How many people are actually going head to head with Google, i.e. making a point to point route server on commercial nationwide data?
“This may be an over simplification but I am relieved to pass the torch of supporting ubiquitous and generalized web mapping applications to the googles, yahoos, mapquests, and microsofts of the world so I can concentrate on the more complex and specialized solution requirements that the suite of commercial applications can’t address.”
I think that is how you “compete” with Google- do something they cannot do and give people an answer that Google cannot give them. You compete by making the topmost layer, not by reproducing the bottom background layer.
I think James is just fishing for views with that title, but maybe its just me.
But his point is well taken. The old traditional method is dead and companies that don’t change are destined to fail.
Investing you time and effort on the Web ADF and big ass AGS servers is not the way forward. The google method of server farms is. Might as well embrace it.
Compete against google? Nope, but for any of us to not try and improve our own offerings is suicide. The Web ADF and ArcGIS is a dinosaur and shouldn’t be used.
It’s striking to me that among the submissions for FOSS4G 2007 presentations include a few talking about WPS (web-based geoprocessing services) and one talking about geoprocessing in a grid computing environment. So there’s something in the air here.
@James: As for The Timoney Group commercializing an open source stack on EC2, my marketing people tells me there’s little money to be made in “GIS”. However, perhaps if we market it as “stuff-you-can-overlay-in-Google-Earth”, the VC spigots would be wide open
@vector: The database bit is a known limitation of EC2 at this time, although there seems to be some very creative MySQL implementations as a workaround. So while it may not be appropriate right now as a persistent repository, it seems to be you could automate the data upload/data processing/download results routine and still have a time- and cost-effective solution.
BT
the DB limitation of EC2 is an issue, the MySQL hack would not perform well in its current implemetation. However, since EC2 has direct access to S3 files, its possible that it could work in the future.
that being said, there is always another way to skin the cat and the Ec2/S3 combination is too good to ignore. GeoServices that expect resource addressable input (think REST) in common formats could be implemented on EC2 right now
cheers brian
“If I were a betting man, I’d put my money on someone putting together a PostGIS/MapServer/GRASS stack on EC2 and selling tile making, interpolations, routing, etc. based on usage.”
Looks like http://www.weogeo.com may be headed in that direction, and might even become a platform for others to do the same with http://weoceo.weogeo.com.
James,
I don’t know if Brian remembers or not, but we (www.WeoGeo.com) were at Location Intelligence in San Francisco showing this exact solution. Adena Schutzberg at Directions Magazine mentioned it briefly here (http://www.directionsmag.com/article.php?article_id=2448&trv=1) and here (http://www.directionsmag.com/podcast.php?id=2469).
The geospatial market will change when skills and content are valued as highly as the infrastructure and sales channels. What Google doesn’t have is the millions of qualified mapping professionals who know their business. They (Google) want your content for free, so that they can sell advertising on top of it. It would be very hard to compete with Google in their sphere.
Another approach is the one we have built on Amazon Web Services (AWS). This (neo)mapping market directly connects content providers with content users, allowing the providers to set the price for their product. Consider it an “Ebay” for mapping skills and content. AWS gives us the reach and scalability to provide people like your reader the ability to do their jobs, be paid for their skills, and participate in this growing internet mapping community.
Now… if we can just get Google to loosen its restrictive licensing and offer service contracts at reasonable prices to small-to-medium sized accounts, we’d all be able to get along.
@Mike – Thanks for the post. I did not see it as I was editing my comment.
One Trackback
[...] developments could especially be important for GIS applications. I know a few other GIS people have blogged about the use of S3 for GIS web apps, and it could really be a boon for people who [...]