Installing programs and loading games, why is this slow?

[Squazzer]

First off, I will apologize for putting this there. I simply just didn't know here to put a question like this. It falls under so many categories.

 

We all know the case:
You are installing a new game, or you are hitting a load-section in your favorite game, and it just takes FOREVER for the program to finish what it is doing. The install process takes too much time, or you hate waiting for loading screens in your game.

 

So what do we do? We do what we always do: troubleshoot the problem, find the bottleneck and upgrades the hardware if possible.

And here comes what I don't get.. Doing this, I have found myself in the situation that I simply just can't find the bottleneck.

 

In most of these cases, my computer just sits there, idle..
CPU is under 50%, the SSD (C:) does close to nothing, the HDD's in raid 0 (G:) havn't been doing anything for a long time. I'm not maxing out my ethernet, and my memory have been sitting at about 7.5Gb for as long as I can remember.

 

Ok, yeah, maybe my total CPU usage is under 50%, but maybe one of my cores is maxing out because all of my applications are trying to hit the same core, right?

No, all of my cores (as shown on my image here below) is doing close to nothing.

 

So my question is: Why does our computers behave like this? Can we do anything about it?

 

 

[TheRandomness]

You know, having half your RAM being used at idle isn't normal, right? What's using so much of it?

[Squazzer]

This is from what uses my memory. But as long as I'm not using 100% of my ram, ram should not be a concern?

[WkdPaul]

Moved to troubleshooting

 

[Nord]

Without a concrete example on where it does happen its hard to tell why exactly but generaly causes are

 

for games:

bad programming / support / engine, to put it simple.

sacrifice performance for modding support or better "compatability" for future updates to the game that add new features (any bethesda game pretty much as example)

bad file compiling / frameworking - lets say you took 50 pictures with your phone and now you look at them at the same time on your PC - so instead of letting your SSD/HDD open/close each one individually you could just compile all of them in a single .jpg and view them there, not worth it if you do it just once or twice but if you do it once a day it would significantely safe you time in the long run - and files required for the game to load pretty much work the same way, one big file to load is better than 1.000 small ones, simply because thats how HDDs/SSDs work better.

 

 

for installation of any programm on windows:

File verification, that wil check if the files arent corrupted after installing.

decompiling files first, usualy used on bigger files to decrease the download size. (Decompiling can be seen as unzipping a zip folder from  a zip folder or basically reverse rendering)

Installing via steam - overload, steam also does file verfications but for that you need to accsess the steam server and maybe sometimes you end up in a que because tons of people want to do the same, hence it takes longer, especially if afterwards a missing file is actually detected and needs to be downloaded again.

Steam is especially special in that case, for example currently Payday 2 has an event where you get to DL a 100~mb patch every day. It takes me about 10~min to get the patch up and running, and with me I mean the automated steam process needs that long even though I got a 50mb/s line and the game aswell as steam are on an SSD. Why? Because Payday 2 is a LUA based game, and LUA is the equivilant of cancer when it comes to gaming. Now add the other stuff from steam as stated above and tada you are back in 1995 in terms of how long stuff takes. Basically I could just download an entire new game from steam and start it in the same time as an 100mb update installs.

 

 

for windows updates and such:

Installing something that is "crucial" to your system to work - or installing it into a component that is being used for your system to work - while your system is running, needs to be done extrely carefully, hence why most update installation will always take ages compared to normal installations and some require a restart. Also most likely they run file vericifation aswell.

 

 

 

What you can do?

Well, become better at computer stuff than everybody who is currently around and invent a more efficient way to do any or all of these things or simply put up with it

[Squazzer]

6 hours ago, Nord said:

for games:

bad programming / support / engine, to put it simple.

... so instead of letting your SSD/HDD open/close each one individually ... one big file to load is better than 1.000 small ones, simply because thats how HDDs/SSDs work better.

I understand this, but in the case with the images, wouldn't I see my harddrives IOPS being maxed out then?

 

6 hours ago, Nord said:

for installation of any programm on windows:

File verification, that will check if the files aren't corrupted after installing.

decompiling files first, usualy used on bigger files to decrease the download size. (Decompiling can be seen as unzipping a zip folder from  a zip folder or basically reverse rendering)

In this case, I would assume that the CPU would spike to 100%, maybe 90 or 80 until the task of decompiling is done. For file comparison, I would assume that there would be an initial load from the harddisk maxing that one out, and then a new spike in the CPU when it's comparing files that are allocated in RAM?

 

6 hours ago, Nord said:

for windows updates and such:

Installing something that is "crucial" to your system to work - or installing it into a component that is being used for your system to work - while your system is running, needs to be done extrely carefully, hence why most update installation will always take ages compared to normal installations and some require a restart. Also most likely they run file vericifation aswell.

I understand this too, but again, I would assume that harddisk or CPU would at least spike?

 

That is the entire case for me here:

Why isn't my computer using 100% percent of its resources. My assumption until now was that the bottlenecking resource was maxed out. But here I can't see that there should be a bottleneck?

 

Actually, I would assume that if I had a supercomputer with extremely fast harddisks and serveral petaflops in CPU power I would be able to install a program, or load a game, within fractions of sections? But to me, it seams there is some parameter that I'm not aware of since I can't find a bottleneck that I can optimize to make it faster?

 

Oh, and for a concrete example: I have experienced this when installing Visual Studio 2015. And I experience it when I play Half Life 2 (the loading sections)

[Nord]

2 hours ago, Squazzer said:

I understand this, but in the case with the images, wouldn't I see my harddrives IOPS being maxed out then?

Thats a case to case scenario and not every file is equal. It would be maxed in a perfect enviroment, like when you were to copy said pictures from HDD 1 to HDD 2.

Because all of this is handled by a single thing that has been around since the dawn of computers and only has 2 "rivals", and thats called windows.

If you apply the same scenario to a game, its very depandant what kind of file it is, what it does for the game, where it needs to go inbetween & where it ends up and especially how the engine of the game handles it further. Like GTA5 pre-loads almost everything and Skyrim/Fallout requires you to load whenever you go throu a door.

 

2 hours ago, Squazzer said:

In this case, I would assume that the CPU would spike to 100%, maybe 90 or 80 until the task of decompiling is done. For file comparison, I would assume that there would be an initial load from the harddisk maxing that one out, and then a new spike in the CPU when it's comparing files that are allocated in RAM?

In case of installing, barely any installation programm will ever use more than either a single thread or a single physical core. I dont know exactly why but thats just how it is.

What I do know is that by default there will always be a "master core" on any CPU, thats one core that will, no matter the scenario, have more work to do than the others. Because he, more or less, needs to instruct the additional cores what to do - to put it in simple terms. (and not all applications even support the use of more than a single core or thread, there are even games out, like Grim Dawn a 2015 game, that only makes use of a single CPU thread or core and the devs sit there and say "thats normal" like they are stuck in 1999 while literally everybody, no matter the system, has FPS drops due to this poor coding and/or ancient engine used - as its the Titan Quest engine)

And either way, for both installation & integrety check, they are not exactly written for speed but to work correctly, always. At least in a perfect scenario and we all know there are way more garbage coders who think they are god out there than ones who actually know what they are doing.

So bottom line is, your hardware is used as much as possible by installations, its usualy the installation itself that bottlenecks.

2 hours ago, Squazzer said:

I understand this too, but again, I would assume that harddisk or CPU would at least spike?

It does, if you use really old hardware. For most of "todays" computers even a single core and a somewhat modern HDD is enough to keep up with smaller installation programms at ease, due to the limitations explained above.

If you want to max your HDD with an installation for example, get a GoG game, like witcher 3, download the entire installation files on your HDD and then make sure the

installation files are on the same HDD as where you are going to install the game, thats when you will encounter HDD limitations, perhaps.

 

2 hours ago, Squazzer said:

But to me, it seams there is some parameter that I'm not aware of since I can't find a bottleneck that I can optimize to make it faster?

Well, learn to code and make better software than is available atm. Its basically the entire "why are you still using java if there is HTML5"-scenario with the same answers.

also you could watch this to understand a bit more about programming, even if you cant yourself:

especially 10:45 "the god object" is relevant to what you are asking.

[Squazzer]

19 hours ago, Nord said:

Well, learn to code and make better software than is available atm. Its basically the entire "why are you still using java if there is HTML5"-scenario with the same answers.

also you could watch this to understand a bit more about programming, even if you cant yourself:

especially 10:45 "the god object" is relevant to what you are asking.

I'm a programmer myself, which is why this question irritates me so much

 

The following images illustrates very well something that irritates me a lot.

None of the cores are working hard, the harddisk does close to nothing, and I'm not using my bandwith to get some data for the installer.

From what I have understood so far the problem *could* be that a core was busy with the install process and the other cores did nothing because the installer didn't utilize other cores.

 

How can even a god-object result in this kind of behavior? I would expect that at least one core was maxed out, and that the fact I was only using one core was the reason the install-process itself could be a bottleneck.

 

I don't understand how bad code can be a bottleneck in a way that results in no bottleneck is visible.

[Nord]

Take a look at this articles:

http://www.howtogeek.com/howto/31204/why-do-application-installs-make-you-reboot-and-close-other-apps/

https://en.wikipedia.org/wiki/Installation_(computer_programs)#Installer

https://en.wikipedia.org/wiki/Software_rot

if you still dont understand why after reading them + what I allready posted, I honestly dont know how to explain it differently other than "thats just how it is, same way the earth is round, it just is."

[Squazzer]

30 minutes ago, Nord said:

Take a look at this articles:

http://www.howtogeek.com/howto/31204/why-do-application-installs-make-you-reboot-and-close-other-apps/

https://en.wikipedia.org/wiki/Installation_(computer_programs)#Installer

https://en.wikipedia.org/wiki/Software_rot

if you still dont understand why after reading them + what I allready posted, I honestly dont know how to explain it differently other than "thats just how it is, same way the earth is round, it just is."

Yeah no, I still don't get how that is connected to the installer not maxing out either the harddisk or at least one of the cores. Sorry

[kirashi]

37 minutes ago, Squazzer said:

Yeah no, I still don't get how that is connected to the installer not maxing out either the harddisk or at least one of the cores. Sorry

It's because no programmer has code that is 100% efficient, same as how there's no power generating device that is 100% efficient.

If we could achieve 100% efficiency of anything, we wouldn't still be humanoid creatures - we'd be mutants by now.

[Squazzer]

12 minutes ago, kirashi said:

It's because no programmer has code that is 100% efficient, same as how there's no power generating device that is 100% efficient.

If we could achieve 100% efficiency of anything, we wouldn't still be humanoid creatures - we'd be mutants by now.

I know that no code 100% efficient, that's the main reason we keep on making faster and faster software on the same hardware.

 

As I have understood it, if a piece of software is 50% efficient it will take me 2 minutes to do the same thing that would take me 1 minutes if the code was 100% efficient?

 

Lets say I write a program whos only purpose in life is to write files to the harddisk.

My CPU can handle much more data than my harddisk, and therefore when my harddisk reaches it max load at 100MB/s my CPU is only using 40% of it's capacity.

Then one of my friends looks at the code, sees I'm a bad programmer, and optimizes the program. Now my CPU only uses 20% of its capacity writing the 100MB/s data to the harddisk.

 

But that's not my case here with the installer. Even In that case my bad code only uses 20% of the CPU, and only writes 50MB/s to the harddisk. Both components can handle more, but they don't utilize the power?

[Squazzer]

1 hour ago, Nord said:

Take a look at this articles:

http://www.howtogeek.com/howto/31204/why-do-application-installs-make-you-reboot-and-close-other-apps/

https://en.wikipedia.org/wiki/Installation_(computer_programs)#Installer

https://en.wikipedia.org/wiki/Software_rot

if you still dont understand why after reading them + what I allready posted, I honestly dont know how to explain it differently other than "thats just how it is, same way the earth is round, it just is."

I wrote me this little program:

Spoiler

 

public void Slow()
{
    var numbers = from num in Enumerable.Range(1,8)
        select FindPrimeNumber(num * 100000000000); //set higher value for more time

    numbers.Dump(); 
}

public long FindPrimeNumber(long n)
{
    int count=0;
    long a = 2;
    while(count<n)
    {
        long b = 2;
        int prime = 1;// to check if found a prime
        while(b * b <= a)
        {
            if(a % b == 0)
            {
                prime = 0;
                break;
            }
            b++;
        }
        if(prime > 0)
        count++;
        a++;
    }
    return (--a);
}

 

It seems that I'm only allowed to use 1/8 of my ressources to run this code. It's like my program gets the qeuavalent of one cores perforamance because I'm running one thread. If I parallelize it I'm allowed to use the full potential of my CPU.

Spoiler

 

public void Slow()
{
    var numbers = from num in Enumerable.Range(1,8).AsParallel()
        select FindPrimeNumber(num * 100000000000); //set higher value for more time

    numbers.Dump(); 
}

public long FindPrimeNumber(long n)
{
    int count=0;
    long a = 2;
    while(count<n)
    {
        long b = 2;
        int prime = 1;// to check if found a prime
        while(b * b <= a)
        {
            if(a % b == 0)
            {
                prime = 0;
                break;
            }
            b++;
        }
        if(prime > 0)
        count++;
        a++;
    }
    return (--a);
}

Is that the key to understanding the problem? My installer is only single-threaded and therefore only allowed to use power equevalent to 1/nth of the cores available? Using 1/nth of the power is not enough to max out any other component, and my CPU is therefore the bottleneck (I know the software is the bottleneck in the end).

 

That would match up with the answer with most votes given here: http://superuser.com/questions/788590/why-does-my-computer-run-slow-if-it-is-not-using-most-of-its-cpu-or-ram

[kirashi]

38 minutes ago, Squazzer said:

I wrote me this little program:

--SNIP--

It seems that I'm only allowed to use 1/8 of my ressources to run this code. It's like my program gets the qeuavalent of one cores perforamance because I'm running one thread. If I parallelize it I'm allowed to use the full potential of my CPU.

--SNIP--

Is that the key to understanding the problem? My installer is only single-threaded and therefore only allowed to use power equevalent to 1/nth of the cores available? Using 1/nth of the power is not enough to max out any other component, and my CPU is therefore the bottleneck (I know the software is the bottleneck in the end).

 

That would match up with the answer with most votes given here: http://superuser.com/questions/788590/why-does-my-computer-run-slow-if-it-is-not-using-most-of-its-cpu-or-ram

That would certainly be part of it. Programs don't automagically scale to use all cores in a system unfortunately.

That's up to the developer, and in some cases, actually results in worse performance than using fewer cores.