Comments on: ESRI Multi-Core and 64-Bit Processors http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/ Geospatial Technology, Web Mapping and Spatial Services Wed, 08 Sep 2010 19:41:17 +0000 hourly 1 http://wordpress.org/?v=3.0.1 By: Mark http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-36308 Mark Wed, 22 Oct 2008 21:16:36 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-36308 <p>Processing imagery or ESRI data sets on 64-bit machine has been a dream of mine ever since I was introduced to the term "64-bit". That was about 10 years ago. I was hoping that ArcGIS 9.1 was going to be 64-bit compliant....how awesome would it be to process a 5 GB image using 16 GB RAM...and ArcGIS is actually able to utilize 16 GB RAM?</p> <p>As for a stable environment on which ESRI should run...when I started using ArcInfo it was only running on a stable platform. Why ESRI departed totally from using UNIX as a viable platform is beyond me. I used ArcInfo on a SUN UNIX server for 6 years and never had to reboot, or even watch it crash, once in that time. When I moved to a Windows platform I was forced to reboot (or simply watch it crash) between 1 and 3 times a day.</p> <p>With Linux available, plus Mac as an industry standard in graphic arts, it's a wonder that ESRI didn't explore other options (i.e. more stable options) of platforms on which to base their product. Having used UNIX, Linux, Mac and PC I have to say that using a Windows based software to process GIS data is, in my opinion, comparably the most counterproductive approach to take. I think this is why Google primarily uses Linux to process and manage all of their spatial data :-).</p> Processing imagery or ESRI data sets on 64-bit machine has been a dream of mine ever since I was introduced to the term “64-bit”. That was about 10 years ago. I was hoping that ArcGIS 9.1 was going to be 64-bit compliant….how awesome would it be to process a 5 GB image using 16 GB RAM…and ArcGIS is actually able to utilize 16 GB RAM?

As for a stable environment on which ESRI should run…when I started using ArcInfo it was only running on a stable platform. Why ESRI departed totally from using UNIX as a viable platform is beyond me. I used ArcInfo on a SUN UNIX server for 6 years and never had to reboot, or even watch it crash, once in that time. When I moved to a Windows platform I was forced to reboot (or simply watch it crash) between 1 and 3 times a day.

With Linux available, plus Mac as an industry standard in graphic arts, it’s a wonder that ESRI didn’t explore other options (i.e. more stable options) of platforms on which to base their product. Having used UNIX, Linux, Mac and PC I have to say that using a Windows based software to process GIS data is, in my opinion, comparably the most counterproductive approach to take. I think this is why Google primarily uses Linux to process and manage all of their spatial data :-) .

]]> By: Multi-cores and 64-bit GIS http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-33474 Multi-cores and 64-bit GIS Wed, 12 Mar 2008 20:35:43 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-33474 <p>[...] hit on this discussion before and with the Developer Summit coming up maybe it is a good time to think about the direction [...]</p> [...] hit on this discussion before and with the Developer Summit coming up maybe it is a good time to think about the direction [...]

]]> By: Jim http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-32124 Jim Thu, 17 Jan 2008 14:26:35 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-32124 <p>The longer I work in an ESRI shop, the more I despise those idiots. For what we pay in software and maintenance fees, their products should be a heck of a lot more stable and a lot more progressive. I hope they cease to exist in the near future. Of course, I'd then have to adapt my skill set, as I've been hitched to nothing but this ESRI train for far too long.</p> The longer I work in an ESRI shop, the more I despise those idiots. For what we pay in software and maintenance fees, their products should be a heck of a lot more stable and a lot more progressive. I hope they cease to exist in the near future. Of course, I’d then have to adapt my skill set, as I’ve been hitched to nothing but this ESRI train for far too long.

]]> By: meeko http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-12669 meeko Sat, 31 Mar 2007 12:37:37 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-12669 <p><i>"I guess we can just use Manifold right? (had to get that in before someone commented about it)."</i></p> <p>thats funny! Though, its also <b>sad<b> when you consider that this company with a $250 GIS seems to have better forward vision than the industry leading GIS vendor.</p> <p>Is it me, or is Manifold seeming to run circles around ESRI when it comes to a technological solution, and future predictions of where to bring the product?</p> <p>That being said, I think ESRI still has the lead because they are actually being used in organizations to solve real world problems, and don't have the speed problems that Manifold has :-)</p> “I guess we can just use Manifold right? (had to get that in before someone commented about it).”

thats funny! Though, its also sad when you consider that this company with a $250 GIS seems to have better forward vision than the industry leading GIS vendor.

Is it me, or is Manifold seeming to run circles around ESRI when it comes to a technological solution, and future predictions of where to bring the product?

That being said, I think ESRI still has the lead because they are actually being used in organizations to solve real world problems, and don’t have the speed problems that Manifold has :-)

]]> By: Matt Eitrem http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2293 Matt Eitrem Mon, 06 Nov 2006 15:12:04 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2293 <p>Quad Cores are here! http://news.com.com/Intels+quad-core+chip+powerful+but+pricey/2100-1041_3-6132033.html</p> Quad Cores are here!
http://news.com.com/Intels+quad-core+chip+powerful+but+pricey/2100-1041_3-6132033.html

]]> By: James Fee http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2292 James Fee Fri, 06 Oct 2006 16:11:32 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2292 <p><i>"I'm looking forward to computing on quad core machines with 4-8GB of usable RAM."</i> But Doug, will ESRI be there with you? ;)</p> “I’m looking forward to computing on quad core machines with 4-8GB of usable RAM.”
But Doug, will ESRI be there with you? ;)

]]> By: Doug http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2291 Doug Fri, 06 Oct 2006 15:59:46 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2291 <p>Joseph, hyperthreding CPUs are not capable of doing twice the work as normal CPUs. Sadly there is an impression that a HT CPU can do twice as CPU intensive work as a non HT CPU because the task manager shows CPU usage at half on a HT machine when it is chugging away.</p> <p>For processor intensive work, a HT machine can actually be a bit slower than a non HT machine... odd but true. In general the performance is the same.</p> <p>scroll down to the charts: http://www.hardwareanalysis.com/action/printarticle/1557/</p> <p>So for x86 the world of multiple threads started with core Duo processors and people who had dual xeon machines. HT wasn't a proper introduction to multiple CPUs and as such most software developers skipped it and consider it to be nothing more than a passing novelty (assuming they have threads that actually do work).</p> <p>Finally for multicore compute intensive tasks I think that you can't separate the issues of x86-64 and multicore. For example: Do you think that you could effectively run four data hungry threads in 1GB Ram? 2GB Ram? My guess is no and that users probably will need a 1GB Ram for each compute intensive thread unless they want to have four cores thrashing the disk as window's VM manager tries to fetch disk pages (which would be very bad).</p> <p>Anyhow... hardware and software are at a good place to move forward. Time to ditch the limitations of single core and 32bit OS's. I'm looking forward to computing on quad core machines with 4-8GB of usable RAM.</p> Joseph, hyperthreding CPUs are not capable of doing twice the work as normal CPUs. Sadly there is an impression that a HT CPU can do twice as CPU intensive work as a non HT CPU because the task manager shows CPU usage at half on a HT machine when it is chugging away.

For processor intensive work, a HT machine can actually be a bit slower than a non HT machine… odd but true. In general the performance is the same.

scroll down to the charts:
http://www.hardwareanalysis.com/action/printarticle/1557/

So for x86 the world of multiple threads started with core Duo processors and people who had dual xeon machines. HT wasn’t a proper introduction to multiple CPUs and as such most software developers skipped it and consider it to be nothing more than a passing novelty (assuming they have threads that actually do work).

Finally for multicore compute intensive tasks I think that you can’t separate the issues of x86-64 and multicore. For example: Do you think that you could effectively run four data hungry threads in 1GB Ram? 2GB Ram? My guess is no and that users probably will need a 1GB Ram for each compute intensive thread unless they want to have four cores thrashing the disk as window’s VM manager tries to fetch disk pages (which would be very bad).

Anyhow… hardware and software are at a good place to move forward. Time to ditch the limitations of single core and 32bit OS’s. I’m looking forward to computing on quad core machines with 4-8GB of usable RAM.

]]> By: Joseph Wallis http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2290 Joseph Wallis Fri, 06 Oct 2006 14:14:54 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2290 <p>ESRI should concentrate on optimizing their code for SMP. Don't worry about 64-bit until version 10. however, 9.2 server products need to be recompiled for 64-bit, since it is there where we need access to larger memory space. Dual Core desktops are starting to become mainstream, but writing multithreaded code would benefit a huge portion of people already running Hyperthreaded boxes (those ARE mainstream). Fortunately Windows has done a fairly good job of using the spare cycles on HT boxes that ArcGIS doesn't take advantage of, but just think if ESRI actually utilized those cycles more efficiently?</p> ESRI should concentrate on optimizing their code for SMP. Don’t worry about 64-bit until version 10. however, 9.2 server products need to be recompiled for 64-bit, since it is there where we need access to larger memory space. Dual Core desktops are starting to become mainstream, but writing multithreaded code would benefit a huge portion of people already running Hyperthreaded boxes (those ARE mainstream). Fortunately Windows has done a fairly good job of using the spare cycles on HT boxes that ArcGIS doesn’t take advantage of, but just think if ESRI actually utilized those cycles more efficiently?

]]> By: Very Spatial &#187; GIS and multi-core computers http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2289 Very Spatial &#187; GIS and multi-core computers Wed, 04 Oct 2006 04:35:06 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2289 <p>[...] ESRI multi-core and 64-bit processors at James Fee GIS Blog [...]</p> [...] ESRI multi-core and 64-bit processors at James Fee GIS Blog [...]

]]> By: Holger http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2288 Holger Fri, 29 Sep 2006 09:35:28 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2288 <p>64 bit on the Desktop may be a minor(?) part, but it becomes a typical Server environment. Windows 2003 Server comes in 64 bit, Oracle (10gR2) comes in 64 bit, but ArcSDE 9.1 still needs a 32 bit client to connect to the database.</p> 64 bit on the Desktop may be a minor(?) part, but it becomes a typical Server environment. Windows 2003 Server comes in 64 bit, Oracle (10gR2) comes in 64 bit, but ArcSDE 9.1 still needs a 32 bit client to connect to the database.

]]> By: Phillip Holmstrand http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2287 Phillip Holmstrand Wed, 27 Sep 2006 18:39:48 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2287 <p>Think there is any chance of getting multi-threading in ArcGIS before these show up?</p> <p>http://news.com.com/2100-1006_3-6119618.html?part=rss&tag=6119618&subj=news</p> Think there is any chance of getting multi-threading in ArcGIS before these show up?

http://news.com.com/2100-1006_3-6119618.html?part=rss&#38;tag=6119618&#38;subj=news

]]> By: Phillip Holmstrand http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2286 Phillip Holmstrand Fri, 22 Sep 2006 17:33:12 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2286 <p>yes yes yes James, you are very right! I'd bring up this issue every week or so until ESRI finally gets it.</p> <p>Computers are not getting any faster, just more parallel.</p> <p>Really what they need to do is support GL or DirectX so hardware acceleration is possible. Then we can finally see graphics at speeds we'd expect to see in 2006, not 1994.</p> <p>Or maybe we'll just wait for somebody else to do it... =o)</p> yes yes yes James, you are very right! I’d bring up this issue every week or so until ESRI finally gets it.

Computers are not getting any faster, just more parallel.

Really what they need to do is support GL or DirectX so hardware acceleration is possible. Then we can finally see graphics at speeds we’d expect to see in 2006, not 1994.

Or maybe we’ll just wait for somebody else to do it… =o)

]]> By: Adam http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2285 Adam Thu, 21 Sep 2006 15:46:36 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2285 <p>I thought for a second the "GIS X" in your graphic said "OS X."</p> <p>I guess that will never happen though, since ArcGIS wouldn't be able to crash OS X every 5 minutes like it can with XP...</p> I thought for a second the “GIS X” in your graphic said “OS X.”

I guess that will never happen though, since ArcGIS wouldn’t be able to crash OS X every 5 minutes like it can with XP…

]]> By: Doug http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2284 Doug Wed, 20 Sep 2006 14:02:18 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2284 <p>I think that it is a bit of an overstatement to say that quad processing is almost here. Yes Intel and AMD have quad core CPUs on the way but dual core isn't that common on the desktop (yet).</p> <p>Of course it will all come in due course- i'm writing this post on my wife's dual mac laptop.</p> I think that it is a bit of an overstatement to say that quad processing is almost here. Yes Intel and AMD have quad core CPUs on the way but dual core isn’t that common on the desktop (yet).

Of course it will all come in due course- i’m writing this post on my wife’s dual mac laptop.

]]> By: BobC http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2283 BobC Wed, 20 Sep 2006 14:01:24 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2283 <p>Actually, with the default windows settings ArcGIS goes thud between 1.3 and 1.5 GB of virtual memory on my box.. even if the actual memory usage is around 500 MB. Usually happens when I'm working with large raster datasets that are tiled into little bittie chunks.</p> <p>A Win32 process is generally limited to 2 GB max address space (This includes MMAP'd files). The rest is used by the OS for page tables and such. There is a kludge that you can include in your boot.ini to increase this process limit to 3 GB, but you also have to flip a bit in the executable file to make it work, and I've not tried it. Fiddling with .exe's often makes license management software go nuts, and I really don't want to deal with it..</p> <p>You can also try relocating .DLL base addresses to leave more contiguous address space, but that's even more of a pain.</p> <p>My usual workaround is to merge the tiles into larger chunks, or try to do that particular bit of processing in gdal or something else with effective memory management.</p> Actually, with the default windows settings ArcGIS goes thud between 1.3 and 1.5 GB of virtual memory on my box.. even if the actual memory usage is around 500 MB. Usually happens when I’m working with large raster datasets that are tiled into little bittie chunks.

A Win32 process is generally limited to 2 GB max address space (This includes MMAP’d files). The rest is used by the OS for page tables and such. There is a kludge that you can include in your boot.ini to increase this process limit to 3 GB, but you also have to flip a bit in the executable file to make it work, and I’ve not tried it. Fiddling with .exe’s often makes license management software go nuts, and I really don’t want to deal with it..

You can also try relocating .DLL base addresses to leave more contiguous address space, but that’s even more of a pain.

My usual workaround is to merge the tiles into larger chunks, or try to do that particular bit of processing in gdal or something else with effective memory management.

]]> By: Matt Eitrem http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2282 Matt Eitrem Wed, 20 Sep 2006 13:41:48 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2282 <p>I agree with James, the lack of Multi-thredding is a big issue, and is only going to grow in the very near future. Quad processing on the average everyday desktop computer is just around the corner. So, you would expect to see this become the norm much sooner on workstations built for GIS professionals. I know I can't wait for a dual processor Core2 machine!</p> I agree with James, the lack of Multi-thredding is a big issue, and is only going to grow in the very near future. Quad processing on the average everyday desktop computer is just around the corner. So, you would expect to see this become the norm much sooner on workstations built for GIS professionals. I know I can’t wait for a dual processor Core2 machine!

]]> By: Petz http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2281 Petz Wed, 20 Sep 2006 09:08:37 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2281 <p>Wasnt 9.0 supposed to be a complete rewrite of ArcGIS in the first place? And why back then didnt they lay the right foundations for multithreaded support and maybe even 64bit operation ? I mean the main benefit for 64bit is the fact that Windows can effectively manage more than 4GB of Ram. I can image in a year or two that more than 4GB of Ram will be a normality for any serious GIS workstation.</p> Wasnt 9.0 supposed to be a complete rewrite of ArcGIS in the first place? And why back then didnt they lay the right foundations for multithreaded support and maybe even 64bit operation ? I mean the main benefit for 64bit is the fact that Windows can effectively manage more than 4GB of Ram. I can image in a year or two that more than 4GB of Ram will be a normality for any serious GIS workstation.

]]> By: Paul Ramsey http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2280 Paul Ramsey Wed, 20 Sep 2006 03:12:54 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2280 <p>Java might finally have an advantage. uDig (Java) has a multi-threaded renderer already and a nice threading API via the Eclipse RCP we use for a workbench/plugin framework. Perhaps this is one of the things that makes Java and .Net "modern" languages?</p> Java might finally have an advantage. uDig (Java) has a multi-threaded renderer already and a nice threading API via the Eclipse RCP we use for a workbench/plugin framework. Perhaps this is one of the things that makes Java and .Net “modern” languages?

]]> By: James Fee http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2279 James Fee Wed, 20 Sep 2006 01:55:38 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2279 <p>I'll be honest, 64 bit is irrelevant for me, but dual core, now that is a different story.</p> <p>Heck multi-threaded is a huge issue for me.</p> I’ll be honest, 64 bit is irrelevant for me, but dual core, now that is a different story.

Heck multi-threaded is a huge issue for me.

]]> By: Doug http://www.spatiallyadjusted.com/2006/09/19/esri-multi-core-and-64-bit-processors/#comment-2278 Doug Wed, 20 Sep 2006 00:24:49 +0000 http://zhun.pair.com/spatiall/blog/?p=1024#comment-2278 <p>Raise your hand if you are running WinXP-64. No commonplace 64 bit OS on the desktop makes it kind of hard to expect that a 64 bit app would have wide appeal.</p> <p>This seems to be something of a problem for lots of apps that would like to go 64 bit (photo and video editing come to mind).</p> <p>Hopefully Vista changes all of this. It looks like OEMs are going to actually bundle 64bit versions of vista with x86-64 chips.</p> Raise your hand if you are running WinXP-64. No commonplace 64 bit OS on the desktop makes it kind of hard to expect that a 64 bit app would have wide appeal.

This seems to be something of a problem for lots of apps that would like to go 64 bit (photo and video editing come to mind).

Hopefully Vista changes all of this. It looks like OEMs are going to actually bundle 64bit versions of vista with x86-64 chips.

]]>