Posted January 21, 2021 Hey guys, long time watcher, first time poster. Im a web developer thats building my next machine. Modern web development (typescrpt, webpack, storybook, jest, cypress, full-stack, docker) is pretty intense on your machine and compilation of typescript and running webpack can really start to bog down your machine. I have 64gigs of RAM and AMD 5900X on order and now looking at the next bottleneck; storage. I am currently rocking a single Samsung 850 PRO 256gb but am wondering if there are any performance gains to be had by upgrading to two in RAID 0 or even Optane? Mostly Typescript compilation involves loading lots (thousands) of small source files (less than 100kb) compiling them all then emitting lots of other small files to disk. Webpack, storybook, jest and all the other menagerie of web-dev tools these days all add huge numbers of file-watchers to your OS too which in turns causes more reads and writes whenever anything changes. Thoughts? Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 15 minutes ago, mikeysee said: is pretty intense on your machine and compilation of typescript and running webpack can really start to bog down your machine. -SNIP- I am currently rocking a single Samsung 850 PRO 256gb but am wondering if there are any performance gains to be had by upgrading to two in RAID 0 or even Optane? Take a look at a proper resource-monitor and the I/O-load on your storage: I'd hazard a guess that you will see the I/O not being the bottleneck. Under Linux, e.g. iotop-c may be used. I do not see why an NVMe-drive would improve things noticeably over a good SATA-drive. A modern SSD can handle hundreds of thousands of I/O-requests per second and, in a compilation like you mentioned, disk bandwidth isn't the limiting factor, but IOPS. In a RAID, the available bandwidth rises but IOPS actually decreases. Hand, n. A singular instrument worn at the end of the human arm and commonly thrust into somebody’s pocket. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 Author 11 minutes ago, WereCatf said: Take a look at a proper resource-monitor and the I/O-load on your storage: I'd hazard a guess that you will see the I/O not being the bottleneck. Under Linux, e.g. iotop-c may be used. Cool, do you have a good suggestion for windows? The reason why I suspect IO to be a bottleneck is because when I run the Typescript compiler on just one of my projects with `--diagnostics` I get: ``` Files: 2892 Lines: 831198 Nodes: 3332589 Identifiers: 911901 Symbols: 785458 Types: 117953 Instantiations: 352615 Memory used: 1086736K I/O read: 0.39s I/O write: 7.94s Parse time: 6.21s Bind time: 1.69s Check time: 5.68s Emit time: 9.52s Total time: 23.10s Done in 23.78s. ``` That emit time taking almost half of the time has me wondering. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 1 minute ago, mikeysee said: Cool, do you have a good suggestion for windows? No, sorry. 1 minute ago, mikeysee said: That emit time taking almost half of the time has me wondering. Pretty easy to test: use a RAM-drive. Hand, n. A singular instrument worn at the end of the human arm and commonly thrust into somebody’s pocket. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 Author I just tried using the disk monitor built into windows Task Manager and it seems like the compilation isnt even registering.. I find that hard to believe. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 38 minutes ago, mikeysee said: Thoughts? Most of the time compilation is limited by processing speed, not your I/O. Given you've got a 5900X on order, I would expect your biggest limitation to be the threadedness of your compiler, rather than any one part of your system. Idk how many threads the typescript compiler can make use of. Edit: by the sounds of it, one. My guess is that you'll see no benefit from either over a good 3.0 NVMe drive. Even a 4.0 drive should be complete overkill. Optane could make a slight difference, maybe, but given it's now been discontinued idk how hard those drives are to find, and I would expect the difference to be of the order of "measurable, but not visible". CPU: i7 4790k, RAM: 16GB DDR3, GPU: GTX 1060 6GB Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 2 minutes ago, mikeysee said: I just tried using the disk monitor built into windows Task Manager and it seems like the compilation isnt even registering.. -SNIP- I find that hard to believe. Given the graph, your SSD is barely hitting 10MB/s bandwidth-usage and it's barely active, just like I said it would. You could try the built-in Resource Monitor, but I do not believe you are even close to being IO-bottlenecked. Hand, n. A singular instrument worn at the end of the human arm and commonly thrust into somebody’s pocket. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 Author Okay cool thanks guys, I have asked the same question here for good measure: https://stackoverflow.com/questions/65820656/what-does-high-i-o-write-and-emit-time-with-typescript-compilation-mean just incase anyone sees this thread in the future and wants to follow along. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 Just now, mikeysee said: Okay cool thanks guys, I have asked the same question here for good measure: https://stackoverflow.com/questions/65820656/what-does-high-i-o-write-and-emit-time-with-typescript-compilation-mean just incase anyone sees this thread in the future and wants to follow along. Just found this thread talking about a similar issue - turns out to be a recursion issue in their code that's causing Emit to take a long time. CPU: i7 4790k, RAM: 16GB DDR3, GPU: GTX 1060 6GB Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 Author 1 minute ago, tim0901 said: Just found this thread talking about a similar issue - turns out to be a recursion issue in their code that's causing Emit to take a long time. Cool good thread, thanks Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted January 21, 2021 2 hours ago, mikeysee said: Cool good thread, thanks SysInternals Process Explorer and Process Monitor should enable you to see how your disk, memory and CPU are performing under which loads, they are part of a set of free utilities for Windows with a lot of configuration, logging and filtering options. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Posted March 15, 2023 On 1/21/2021 at 2:05 PM, tim0901 said: Most of the time compilation is limited by processing speed, not your I/O. Given you've got a 5900X on order, I would expect your biggest limitation to be the threadedness of your compiler, rather than any one part of your system. Idk how many threads the typescript compiler can make use of. Edit: by the sounds of it, one. My guess is that you'll see no benefit from either over a good 3.0 NVMe drive. Even a 4.0 drive should be complete overkill. Optane could make a slight difference, maybe, but given it's now been discontinued idk how hard those drives are to find, and I would expect the difference to be of the order of "measurable, but not visible". Sorry for the late reply, but I believe I have to mention the MB/s numbers didn't necessarily mean the disk was sitting idle. If you benchmarked an SSD (or looked up the benchmarks made by linustechtips), you would find the speeds reading and writing small files always much lower than the sequancial speed a disk can get to. For example, I am using a Samsung 980 pro. Its full speed read (sequancial) is >6000MB/s, however, its 4k performance is usually around 70MB/s, about 1/100 of its sequancial read speed. In the case of software compilation, all the source files are tiny. With my typescript project for instance, I have 20k files that sum up to ~3GB of space. Most of of the source code files are smaller than 4k. From my experience, they are usually 2kb each. So when they are compiled, you would typically see <70MB/s when the drive is already saturated. The disk is just unable to possibly get any higher than that. According to Intel's own report regarding compiler performance, they quoted a 25% performance increase over 850 EVO. https://cdrdv2-public.intel.com/760271/nsg-client-solution-kit-gaming-solution-overview.pdf With a high-end model like 980 pro, it may get better than 850 EVO with a few percents, but not a lot, since its 4k performance remains fairly close (~50MB/s 850 EVO, ~70MB/s 980 pro). The advancement of NAND SSD these years are all invested on sequancial reads and writes. In contrast, an optane drive like Intel 905p can get 300-400MB/s 4k speed on a good CPU. It's in fact bottlenecked by the CPU and NVMe/PCIe protocol here. Link to comment Share on other sites More sharing options... Link to post Share on other sites More sharing options...
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now