Geographic Information Systems (GIS) Extract Translate and Load (ETL) Workstation

[Riche McD]

Im looking for advice and help to build/Spec a system for work.

 

Budget (Euro): 8000..ish

Country: Ireland

Games, programs or workloads that it will be used for: Purely for Productivity. I work as a Geographic Information Systems (GIS) Specialist, working with both Vector and Raster datasets. My role generally involves working with large datasets, for example Rasters in excess of 40GB and up to 80GB National Datasets. 

Other software used include:

QGIS https://qgis.org/en/site/

FME https://www.safe.com/

ArcGIS https://www.arcgis.com/index.html

Pix4D https://www.pix4d.com/

OSGeo4W https://www.osgeo.org/

Postgresql (PostGIS)

 

Current System: 

OS Name    Microsoft Windows 7 Professional

System Manufacturer    Dell Inc.

System Model    Precision Tower 7810

System Type    x64-based PC

Processor    Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz, 1600 Mhz, 6 Core(s), 6 Logical Processor(s) x 2

Graphics     NVIDIA Quadro K2200 x 2

Installed Physical Memory (RAM)    64.0 GB
Drives   

                   C Drive     Size    3.64 TB (4,000,645,640,192 bytes)

                   D Drive     Size    452.07 GB (485,401,554,944 bytes)

 

 

I Believe both are mechanical not SSD. The current system is capable, but slow, its also reaching the max of its capabilities in the generation of the 40 - 80GB Raster datasets.

 

Additional:

What I am looking for is a powerful Extract Translate and Load (ETL) system, bulk of work is carried out using FME or using GDAL/OCR Libraries directly from command line. I need a system which has as few bottle necks as possible and efficiently balances the workload across the system, also multi tasking would be great, running two processes simultaneously would be great.

 

Any help or guidance welcome,

Regards,

 

RicheMCD



 

 

 

[Radium_Angel]

4 minutes ago, Riche McD said:

I need a system which has as few bottle necks as possible and efficiently balances the workload across the system, also multi tasking would be great, running two processes simultaneously would be great.

Is your GIS program multi-core aware? In other words, how well does it tax the twin Xeons you have?

And if you are running spinner drives, you will be amazed at the time saved going m.2 nVME or SSD.

[Riche McD]

No, I can set values for it to utilise additional All Cores in processing, but, not currently for rendering.

 

[Riche McD]

GDAL allows for use of all cores when processing GEOTiff, further efficiencies can be built into the rasters for rendering purposed on Desktop GIS ie. QGIS, but it takes more processing time to build in the Tiling, Compression and adding Overviews

 

[Radium_Angel]

Just now, Riche McD said:

GDAL allows for use of all cores when processing GEOTiff, further efficiencies can be built into the rasters for rendering purposed on Desktop GIS ie. QGIS, but it takes more processing time to build in the Tiling, Compression and adding Overviews

 

BTW, quote me so I get a notification that you've replied.

 

So the question I have for you then is this:

Where's the slowdown in your current system?

Loading the data, or crunching it?

[Riche McD]

1 minute ago, Radium_Angel said:

BTW, quote me so I get a notification that you've replied.

 

So the question I have for you then is this:

Where's the slowdown in your current system?

Loading the data, or crunching it?

 

Both, I recently completed a 40GB raster with compression, and tiling, implementing the all cores option on GDAL. It took 29 hours to complete.

I can be required to have 3 or more of these images loaded onto my Desktop GIS software (QGIS) at any one time, rendering these can take a while upwards of a minute.

 

[Radium_Angel]

6 minutes ago, Riche McD said:

 

Both, I recently completed a 40GB raster with compression, and tiling, implementing the all cores option on GDAL. It took 29 hours to complete.

I can be required to have 3 or more of these images loaded onto my Desktop GIS software (QGIS) at any one time, rendering these can take a while upwards of a minute.

 

Ok, so SSDs or m.2 nVME will help there.

How hard is the GFX cards hit by doing the data crunching?

[Riche McD]

2 minutes ago, Radium_Angel said:

Ok, so SSDs or m.2 nVME will help there.

How hard is the GFX cards hit by doing the data crunching?

Not very, seems like a wasted resource. I saw on the new AMD processors they have the capability to tap into unused resources on the GFX cards, its probably something to consider. Im looking with a view to build from scratch.

[Radium_Angel]

Just now, Riche McD said:

Not very, seems like a wasted resource. I saw on the new AMD processors they have the capability to tap into unused resources on the GFX cards, its probably something to consider. Im looking with a view to build from scratch.

I understand, I'm just trying to get a handle on where to put the bulk of your money.

Those twin Xeons are very slow, in terms of raw speed. I have a T5610 that came with stock 12 cores in 2 CPUs, but only at 2Hgz. Replacing them with ones that ran at 3.5Ghz make a huge difference.

If the software is killing your CPUs, then Ghz is king, and go looking into something super speedy, like a threadripper, or if the software has an affinity for dual CPUs, EPYC is an option (and I say this as a die-hard fan of Xeons....AMD is really making some solid leaps here)

 

Do you want to build your own system, or go with another pre-built workstation like a Precision?

 

[Riche McD]

5 minutes ago, Radium_Angel said:

I understand, I'm just trying to get a handle on where to put the bulk of your money.

Those twin Xeons are very slow, in terms of raw speed. I have a T5610 that came with stock 12 cores in 2 CPUs, but only at 2Hgz. Replacing them with ones that ran at 3.5Ghz make a huge difference.

If the software is killing your CPUs, then Ghz is king, and go looking into something super speedy, like a threadripper, or if the software has an affinity for dual CPUs, EPYC is an option (and I say this as a die-hard fan of Xeons....AMD is really making some solid leaps here)

 

Do you want to build your own system, or go with another pre-built workstation like a Precision?

 

Not sure yet, we have a lot of work coming in the new year, which will have the machine hopping 24-7. I want to get a specification together that I can go to my boss with to say, this is where we should be looking at going with the workstations. Building in-house is an option, but with current procurement rules, maybe the prebuilt might be the way to go. ideal situation for me would be to have 

1. Custom all bells and whistles
2. Dell Pre-built Option

If that makes sense? 

[Radium_Angel]

5 minutes ago, Riche McD said:

Not sure yet, we have a lot of work coming in the new year, which will have the machine hopping 24-7. I want to get a specification together that I can go to my boss with to say, this is where we should be looking at going with the workstations. Building in-house is an option, but with current procurement rules, maybe the prebuilt might be the way to go. ideal situation for me would be to have 

1. Custom all bells and whistles
2. Dell Pre-built Option

If that makes sense? 

Ok, so you work for someone.

Here's my suggestion:

Go to Dell, talk with them on the phone (don't deal with their level 1 guys, go business or gov't sales) 

Get a price quote for a new Precision based on speed being priority (Ghz) and SSDs (they'll all have that) 

Don't forget with a business workstation, you get onsite support and a warranty, which is something hard to dismiss, especially if things go wrong with the system.

 

Bring the quote back here, and see if the LTT community can build faster for less.

 

Keep in mind tho, this is a workstation, reliability and uptime is a critical factor here, it's one of the main reasons I ditched "home systems" and went "workstation"

 

Alternately, go talk with the software vendors and ask what they would recommend for specs, and bring those specs back here, and let's see what we can do with them.

[Riche McD]

2 minutes ago, Radium_Angel said:

Ok, so you work for someone.

Here's my suggestion:

Go to Dell, talk with them on the phone (don't deal with their level 1 guys, go business or gov't sales) 

Get a price quote for a new Precision based on speed being priority (Ghz) and SSDs (they'll all have that) 

Don't forget with a business workstation, you get onsite support and a warranty, which is something hard to dismiss, especially if things go wrong with the system.

 

Bring the quote back here, and see if the LTT community can build faster for less.

 

Keep in mind tho, this is a workstation, reliability and uptime is a critical factor here, it's one of the main reasons I ditched "home systems" and went "workstation"

 

Alternately, go talk with the software vendors and ask what they would recommend for specs, and bring those specs back here, and let's see what we can do with them.

Thanks so much for your advice and for taking the time to respond. A lot to think about. 

[Radium_Angel]

1 minute ago, Riche McD said:

Thanks so much for your advice and for taking the time to respond. A lot to think about. 

It is, I'm working on Dell's Precision site now, looking at options, and can easily hit 8K, but if we can scavenge some of the parts from your old system, we can bring the price down a bit