Asrock Rack and SuperMicro announces Xeon Phi systems

[patrickjp93]

16 minutes ago, IvantheDugtrio said:

The wrapper for the pipeline is python but the actual HPC is a mix of C and Java. I'll be working primarily on the C code.

 

I'll also see if I can configure the pipeline to run certain tools on specific nodes of this asymmetrical torque cluster (ported C tools on the Xeon Phi, everything else on the regular Xeons).

Even though it's a Python wrapper, you have to use double the memory and spend the time massaging the data back and forth. For Pete's sake why is the bioinformatics community still dealing with the shit language (for HPC at least) that is Python in the face of C++ 11, let alone 14?! And Java is not much better, but at least it can natively pass data to C/C++ routines without copying.

[Curufinwe_wins]

35 minutes ago, patrickjp93 said:

Even though it's a Python wrapper, you have to use double the memory and spend the time massaging the data back and forth. For Pete's sake why is the bioinformatics community still dealing with the shit language (for HPC at least) that is Python in the face of C++ 11, let alone 14?! And Java is not much better, but at least it can natively pass data to C/C++ routines without copying.

I mean.... the nuclear community is still dealing with a massive proportion of Fortran code....

[patrickjp93]

4 minutes ago, Curufinwe_wins said:

I mean.... the nuclear community is still dealing with a massive proportion of Fortran code....

Fortran was designed for high performance scientific computing. It's just not nice for doing it in a modern way where coding can be done much faster in C++ while having the same or better efficiency. That's why Intel and IBM still provide Fortran compilers for existing codebases.

[Curufinwe_wins]

11 minutes ago, patrickjp93 said:

Fortran was designed for high performance scientific computing. It's just not nice for doing it in a modern way where coding can be done much faster in C++ while having the same or better efficiency. That's why Intel and IBM still provide Fortran compilers for existing codebases.

I am well aware. Also by Fortran, I mean FORTRAN 77. I have the 80 pounds of original punch card code sitting around for one of those programs.

 

Unfortunately the desire to maintain perfect backwards compatibility makes moving to a new language a bit more of a large upfront time investment. And most of these codes aren't actively maintained by more than a single person.

[patrickjp93]

42 minutes ago, Curufinwe_wins said:

I am well aware. Also by Fortran, I mean FORTRAN 77. I have the 80 pounds of original punch card code sitting around for one of those programs.

 

Unfortunately the desire to maintain perfect backwards compatibility makes moving to a new language a bit more of a large upfront time investment. And most of these codes aren't actively maintained by more than a single person.

Bring in a newbie who loves C++ and put them to work doing code conversion. I just spent a week porting 2500 lines of Java to C++. It can be done.

[Curufinwe_wins]

5 minutes ago, patrickjp93 said:

Bring in a newbie who loves C++ and put them to work doing code conversion. I just spent a week porting 2500 lines of Java to C++. It can be done.

We have a little bit over 100,000 lines of Fortran to convert (for that program I mentioned), and literally backwards compatibility with previous inputs needs to be flawless.

 

I'm sure it's doable, just no one wants to devote resources to it... Since technically speaking it still works...

[patrickjp93]

18 minutes ago, Curufinwe_wins said:

We have a little bit over 100,000 lines of Fortran to convert (for that program I mentioned), and literally backwards compatibility with previous inputs needs to be flawless.

 

I'm sure it's doable, just no one wants to devote resources to it... Since technically speaking it still works...

If by inputs you mean command line arguments, that's already there. If it's a matter of file formats being read, you can customize a stream iterator for that. If anything else, it's exactly the same.

[IvantheDugtrio]

On 9/16/2016 at 7:53 PM, patrickjp93 said:

Even though it's a Python wrapper, you have to use double the memory and spend the time massaging the data back and forth. For Pete's sake why is the bioinformatics community still dealing with the shit language (for HPC at least) that is Python in the face of C++ 11, let alone 14?! And Java is not much better, but at least it can natively pass data to C/C++ routines without copying.

Lol the wrapper is just passing commands and logging output. The performance impact isn't there. Python is used as an easy way to manage the versions of the actual tools and libraries used via anaconda. The wrapper also provides a convenient way to use cluster computing for all of the programs run under it. All of the real HPC is done by the C, C++, and Java programs. 

 

Java is used in HPC for its portability across different architectures. The real performance comes from the JRE and its ability to efficiently use system resources. Some of our Java-based tools such as the GATK support AVX instructions for further acceleration. Using Java also cleans up the code a lot compared with C++ for complex programs such as variant callers that have to recognize a multitude of mutations in samples. I'm not aware of any C-based variant caller out there simply because of code complexity reasons. The tools that do use C/C++ are the aligners, compression/decompression tools, and stream editors that rely on a single algorithm for processing data. 

[patrickjp93]

10 minutes ago, IvantheDugtrio said:

Lol the wrapper is just passing commands and logging output. The performance impact isn't there. Python is used as an easy way to manage the versions of the actual tools and libraries used via anaconda. The wrapper also provides a convenient way to use cluster computing for all of the programs run under it. All of the real HPC is done by the C, C++, and Java programs. 

 

Java is used in HPC for its portability across different architectures. The real performance comes from the JRE and its ability to efficiently use system resources. Some of our Java-based tools such as the GATK support AVX instructions for further acceleration. Using Java also cleans up the code a lot compared with C++ for complex programs such as variant callers that have to recognize a multitude of mutations in samples. I'm not aware of any C-based variant caller out there simply because of code complexity reasons. The tools that do use C/C++ are the aligners, compression/decompression tools, and stream editors that rely on a single algorithm for processing data. 

That's not true. Run the program in raw C++ and you'll see a difference in memory usage that's nearly a factor of 2x.

 

The portability argument is BS. You can compile C++ on any architecture as long as you don't use OS-specific libraries or system calls or do in-line assembly of the wrong architecture. And JREs and efficiency should never be in the same sentence. They're never efficient.

 

HAHAHAHA! Cleans up the code? BS. You can write the same program in fewer lines and fewer words per line using C++ 11 and 14.

 

Despite the fact I don't know what you mean by variant callers, if you mean overloaded function calls with varying variables, that can be done easily using variadic templates under C++ 11 and 14.

 

All of your tools should be in C++ both for productivity and speed reasons.

[lukart]

Wasnt Intel Phi a reply to nvidias super computing GPUs? 

I wonder for those who know this market how they are doing in terms of market share.

[patrickjp93]

3 hours ago, lukart said:

Wasnt Intel Phi a reply to nvidias super computing GPUs? 

I wonder for those who know this market how they are doing in terms of market share.

Intel is stealing some market share, but the majority of it is still in Nvidia's hands. And the history of the Xeon Phi is a bit more complicated than that. Project Larabee started out as Intel's attempt to enter the dGPU market. It licensed a ton of tech from Nvidia and ran a collaborative project spending more money than Nvidia and AMD/ATI had combined since they were founded (on GPU research). A couple months before launch was due, Nvidia saw what a threat it would be and yanked a number of fundamental patents, buying out of parts of the contract. It basically made it impossible to sell the product as-is. Intel quickly pivoted and turned it into an x86-native coprocessor.

 

Now with Knight's Landing you can get it as a coprocessor or as a host CPU. The compute density per node is much higher than any IBM, AMD, or ARM system.

[NumLock21]

CPU size comparison and new type of mounting

https://www.servethehome.com/big-sockets-look-intel-lga-3647/

 

 

[Space Reptile]

On 24.8.2016 at 6:48 AM, Ryan_Vickers said:

That is a massive socket...

eh , G34 was bigger (or is atleast the same size :P)