A Probability Simulator.

[TheAmazingDanny]

So, I want to post this here because I know Linus likes to review different CPUs and systems for productivity.

 

I'm making a tool that will allow a computer to simulate probability by using C#'s built-in RNG to simulate many trials of many experiments with a varying number of outcomes (default is 2, to simulate a "coin flip"). Experiments have end conditions, and the ones my program allow for are for an amount of time in seconds, an amount of repetitions, or a pattern to be hit.

 

This tool can export to CSV, an HTML document (still in the works), and LaTeX. I'm using it for a final in my statistics class, but an average amount of repetitions per second can be gathered from the output LaTeX document (with standard deviation), and, last night, I did thirty trials of running the program for 60 seconds, giving an end result of around 60 million repetitions per second on my Ryzen 3 1300X (at the stock frequency of 3.5 GHz, I can't afford a better cooler).

 

I'm looking for a way to contact the Linus Media Group and see if they would be interested in using this in the future. Not all benchmark output can be compiled into a PDF! Attached is the PDF of the run of it I did last night and a screenshot of the title page.

Results.pdf

[straight_stewie]

8 hours ago, TheAmazingDanny said:

This tool can export to CSV, an HTML document (still in the works), and LaTeX. I'm using it for a final in my statistics class, but an average amount of repetitions per second can be gathered from the output LaTeX document (with standard deviation), and, last night, I did thirty trials of running the program for 60 seconds, giving an end result of around 60 million repetitions per second on my Ryzen 3 1300X (at the stock frequency of 3.5 GHz, I can't afford a better cooler).

What exactly is this benchmarking?

[honna1612]

First of all. C# System.Random is very slow. Look Up XORShift C#. Its about 60x faster on a x64 CPU. Also why not 8 Threads on 4 Cores? Hyperthreading gives 5-15% more performance for free you know.  

 

And why the hell would LMG be interested in a simple Coin flip program? 

[TheAmazingDanny]

16 hours ago, straight_stewie said:

What exactly is this benchmarking?

I'm sort of "benchmarking" by simulating how many repetitions can be done per second, and calculating standard deviation over many trials to see how well it can keep that performance.

[TheAmazingDanny]

6 hours ago, honna1612 said:

First of all. C# System.Random is very slow. Look Up XORShift C#. Its about 60x faster on a x64 CPU. Also why not 8 Threads on 4 Cores? Hyperthreading gives 5-15% more performance for free you know.  

 

And why the hell would LMG be interested in a simple Coin flip program? 

It benchmarks how many "coin flips" can be done in a second. Cinebench benchmarks, to my understanding, how fast it can render a huge image with the CPU... how is this any different?

 

I'll have to check out XOR Shift eventually, but this is the start of something great.

 

Also, to answer your point about hyper-threading? My Ryzen 3 1300X doesn't support hyper-threading like an i3 pre-Coffee Lake would be 2C/4T. Ryzen 3 is 4C/4T like an old i5 was. The Ryzen 5 is 4C/8T.

[TheAmazingDanny]

7 hours ago, honna1612 said:

First of all. C# System.Random is very slow. Look Up XORShift C#. Its about 60x faster on a x64 CPU. Also why not 8 Threads on 4 Cores? Hyperthreading gives 5-15% more performance for free you know.  

 

And why the hell would LMG be interested in a simple Coin flip program? 

Update: I swapped my implementation to the one shown at https://www.programmingalgorithms.com/algorithm/xor-shift?lang=C%23 and I'm running benchmarks now. I did change the implementation to use the current date and time as a seed with each trial. Thank you for showing me this!

[tikker]

1 hour ago, TheAmazingDanny said:

It benchmarks how many "coin flips" can be done in a second. Cinebench benchmarks, to my understanding, how fast it can render a huge image with the CPU... how is this any different?

 

I'll have to check out XOR Shift eventually, but this is the start of something great.

 

Also, to answer your point about hyper-threading? My Ryzen 3 1300X doesn't support hyper-threading like an i3 pre-Coffee Lake would be 2C/4T. Ryzen 3 is 4C/4T like an old i5 was. The Ryzen 5 is 4C/8T.

Mostly that coin flips are simple, aren't really tied to any real world performance and are the same thing just repeated X times, i.e. you're really only testing the simplest thing your CPU can do: generate a number. Cinebench is 3D rendering, which is a much more intensive load in the sense of what the CPU has to do. It also helps that it has been an industry standard for a while now, so there is a large database of existing scores (this will be the biggest point of resistance for a new benchmark I think).

 

Still cool to see someone building their own type of benchmark.

 

[TheAmazingDanny]

On 4/28/2018 at 5:21 PM, tikker said:

Mostly that coin flips are simple, aren't really tied to any real world performance and are the same thing just repeated X times, i.e. you're really only testing the simplest thing your CPU can do: generate a number. Cinebench is 3D rendering, which is a much more intensive load in the sense of what the CPU has to do. It also helps that it has been an industry standard for a while now, so there is a large database of existing scores (this will be the biggest point of resistance for a new benchmark I think).

 

Still cool to see someone building their own type of benchmark.

 

While a probability simulator is simpler in execution and depends more on the ALU, it also does shine in the fact that it shows how well that performance can be kept. Say, you have a laptop that thermal throttles - that'll cause a higher standard deviation since there will be an outlier with many less reps/second, no?

 

Put it this way. I run a test for 12 hours. (That's what I was doing while I wasn't responding; I ran 12 hours of that simulation.) Using my aforementioned bench and the swap on the RNG algorithm, my Ryzen 3 1300X, now OC'd to 3.7 GHz, can strike 78.5 million reps/second, with a standard deviation of around 500,000.

 

What does that show? It shows a more of a "what you can expect" for performance, as it's a mean of all trials, and standard deviation shows fluctuation. It shows that, using a bell curve, it can do 78.5 million reps/sec +- 500k, roughly 68% of the time.

 

It's more of why we switched to 1% and .1% lows rather than lowest value; standard deviation shows how well it can keep that value. I may modify this program to include max/min of the trials as well.

[wasab]

If you want to test throttling, I doubt the algorithm you wrote can stress a cpu to the max just by doing coin toss. You need to code a hardcore heat virus to really push a limit. Nothing can beat prime95 at that. 

[TheAmazingDanny]

2 hours ago, wasab said:

If you want to test throttling, I doubt the algorithm you wrote can stress a cpu to the max just by doing coin toss. You need to code a hardcore heat virus to really push a limit. Nothing can beat prime95 at that. 

...Prime95 is a program meant to find Mersennes primes (that is, a program made to find prime numbers that take the form 2ⁿ - 1). See their product page about it. It's not a "Hardcore head virus" as you say; rather; it does have a goal. It runs a cpu busy workload; that is, something that takes the form of:

 

while (some condition that will take a while to compute) {
	// perform an intensive task
}

In this case, I use:

DateTime endTime = DateTime.UtcNow.AddSeconds((double) ExpManager.sharedInstance.endCondition.seconds);
while (DateTime.UtcNow.CompareTo(endTime) <= 0)
{
    storeResult(diceRoll());
}

So, while that amount of time didn't pass, the CPU will keep looping back to the beginning instruction. I will add that this is being run on each thread of the CPU, making this a suitable "heat virus" from your definition.

The group recently found that, thanks to the help of more modern CPUs and people leaving this sort of thing running, 2^77,232,917 - 1 is a prime number, and that all exponents below 78 million have been tested at least once.

That does give the CPU some number crunching work, but this has other usefulness for those wanting to run a simulation of millions of coin flips, or any experiment, really, with a variable amount of outcomes.

[tikker]

5 hours ago, TheAmazingDanny said:

So, while that amount of time didn't pass, the CPU will keep looping back to the beginning instruction. I will add that this is being run on each thread of the CPU, making this a suitable "heat virus" from your definition.

Giving the max and average temperature might be nice output as well. What @wasab means to say, I think, is that coin flips aren't exactly a heavy load. It's just pulling a number from a distribution, not calculation needs to be done versus the factoring and FFT-ing that P95 does. While perhaps not designed as a "heat virus" specifically, Prime95 stresses CPU big time and generates a lot of heat because of that (also because of AVX); much more than a coin flip can do.

Still it's nice to do this on a larger scale. I got it as practice material during my undergrad as programming exercise and introductory statistics (e.g. binomial distribution, coin fairness), but because the sample is large enough quickly scaling it up was never considered.

8 hours ago, TheAmazingDanny said:

While a probability simulator is simpler in execution and depends more on the ALU, it also does shine in the fact that it shows how well that performance can be kept. Say, you have a laptop that thermal throttles - that'll cause a higher standard deviation since there will be an outlier with many less reps/second, no?

True, but it's much more useful to know if your laptop throttles during cinebench than during coin flips, since it's (arguably, can of course differ) more representative of a real-world load. That's also why Prime95 is sometimes not weighted that significantly, since it's such an artificially high load, not representing day to day use.

Also I don't quite get what your program actually does. It looks like your simulating an N-sided coin? What do you mean by "simulate probability". AFAIK the random number generators in a computer are (or at least try to be) uniform, such that there is no real bias towards any value?

[wasab]

4 hours ago, tikker said:

 

Also I don't quite get what your program actually does. It looks like your simulating an N-sided coin? What do you mean by "simulate probability". AFAIK the random number generators in a computer are (or at least try to be) uniform, such that there is no real bias towards any value?

He looks like to be using java so he is probably seeding the pesudo random generator with system time if he is using the standard java random library. If ran for more than a day, those “random” numbers will just repeat themselves 

 

Edit: that was c#! My bad haha. Still, he is likely to be using system time as the seed. I can’t think of any other better seed really. 

[DtrollMC]

8 hours ago, wasab said:

He looks like to be using java so he is probably seeding the pesudo random generator with system time if he is using the standard java random library. If ran for more than a day, those “random” numbers will just repeat themselves 

 

Edit: that was c#! My bad haha. Still, he is likely to be using system time as the seed. I can’t think of any other better seed really. 

I don't think it would repeat after a day. System time records the DATE and TIME, using both would constantly give new numbers.

[wasab]

22 minutes ago, DtrollMC said:

I don't think it would repeat after a day. System time records the DATE and TIME, using both would constantly give new numbers.

Is that so. Well, as long as the seed doesn’t repeat, there shouldn’t be a pattern. Mouse position is another way to seed a random although that won’t work if user doesn’t move the mouse at all haha.

 

[tikker]

37 minutes ago, DtrollMC said:

I don't think it would repeat after a day. System time records the DATE and TIME, using both would constantly give new numbers.

 

12 minutes ago, wasab said:

Is that so. Well, as long as the seed doesn’t repeat, there shouldn’t be a pattern. Mouse position is another way to seed a random although that won’t work if user doesn’t move the mouse at all haha.

 

It's C#, so the .NET API. After a day I'd be more worried about periodicity though, as the RNG's implementation seems to be a little broken. Though that probably doesn't matter for the benchmark, just skew the statistics a bit.

[DtrollMC]

1 minute ago, tikker said:

 

It's C#, so the .NET API. After a day I'd be more worried about periodicity though, as the RNG's implementation seems to be a little broken. Though that probably doesn't matter for the benchmark, just skew the statistics a bit.

I really know nothing about random numbers at all, just when I've pondered it before I recognized System time is a number that's never the same at two moments.

[tikker]

5 minutes ago, DtrollMC said:

I really know nothing about random numbers at all, just when I've pondered it before I recognized System time is a number that's never the same at two moments.

That's true, so it's useful to use as a seed in that way. The problem is that random number generators aren't completely random (hence why they're called pseudo-RNGs). Let's not derail this further though  

[Mira Yurizaki]

The thing I'm canting my head about this with regards to using this as a benchmark is what exactly is being exercised here? I'm assuming here, but I'm assuming the gist of the code is basically:

/* Assume some timer event will flip "runBenchmark" to false */
while(runBenchmark){
  headsTailsCounter[rng.next(0,1)]++;
}

Granted maybe the PRNG is heavy in terms of execution, but from a higher level point of view, I'm not sure what this data could be useful for when trying to gauge performance in some other task.

 

It'd be something like:

/* Assume:
- Some timer event will flip "runBenchmark" to false 
- Counter is a 64-bit uint
- Benchmark doesn't ever run long enough for counter to overflow, unless that's the intent.
*/
while(runBenchmark){
  counter ++;
}

And though I understand C# doesn't work like C when it comes to compiling code, I already take issue with either one because I know that it will never exercise the processor that well. Why? Well let's just take the second example. It's a single instruction (it should compile to something like ADD counter 1), but modern processors are capable of instruction level parallelism. Given that Zen and Coffee Lake have four ALUs for ILP, the second task is potentially wasting 75% of how many ALUs there really are in the execution unit. So really, the second benchmark should be:

/* Assume:
- Some timer event will flip "runBenchmark" to false 
- The counters are a 64-bit uint
- Benchmark doesn't ever run long enough for counter to overflow, unless that's the intent.
*/
while(runBenchmark){
  counter1 ++;
  counter2 ++;
  counter3 ++;
  counter4 ++;
}

Of course, this is also making an assumption that the CPU will actually invoke ILP on this.

 

EDIT: On a side note, if you're doing this:

/* Assume some timer event will flip "runBenchmark" to false */
while(runBenchmark){
  int coinSide = rng.next(0,1);
  if (coinSide == 0)
    headsCounter ++;
  else
    tailsCounter ++;
}

Performance may be determined by the branch predictor performance or design, rather than how well the processor can actually perform tasks.

Edited April 30, 2018 by M.Yurizaki

[TheAmazingDanny]

14 hours ago, tikker said:

Giving the max and average temperature might be nice output as well. What @wasab means to say, I think, is that coin flips aren't exactly a heavy load. It's just pulling a number from a distribution, not calculation needs to be done versus the factoring and FFT-ing that P95 does. While perhaps not designed as a "heat virus" specifically, Prime95 stresses CPU big time and generates a lot of heat because of that (also because of AVX); much more than a coin flip can do.

Still it's nice to do this on a larger scale. I got it as practice material during my undergrad as programming exercise and introductory statistics (e.g. binomial distribution, coin fairness), but because the sample is large enough quickly scaling it up was never considered.

True, but it's much more useful to know if your laptop throttles during cinebench than during coin flips, since it's (arguably, can of course differ) more representative of a real-world load. That's also why Prime95 is sometimes not weighted that significantly, since it's such an artificially high load, not representing day to day use.

Also I don't quite get what your program actually does. It looks like your simulating an N-sided coin? What do you mean by "simulate probability". AFAIK the random number generators in a computer are (or at least try to be) uniform, such that there is no real bias towards any value?

It does simulate flipping an N-sided coin. I'm using an end condition where, rather than doing a set amount of flips, it keeps "flipping" a coin until one minute has passed from the start of the experiment. Prime95 does do actual calculation with those values, and I'll give it that, however, this workload could stress an ALU.

[TheAmazingDanny]

13 hours ago, wasab said:

He looks like to be using java so he is probably seeding the pesudo random generator with system time if he is using the standard java random library. If ran for more than a day, those “random” numbers will just repeat themselves 

 

Edit: that was c#! My bad haha. Still, he is likely to be using system time as the seed. I can’t think of any other better seed really. 

I think I answered this in a previous comment; I was using System.Random before another comment mentioned to XORShift, and I went with that since it's much faster.

5 hours ago, DtrollMC said:

I don't think it would repeat after a day. System time records the DATE and TIME, using both would constantly give new numbers.

You are correct - if time is a number, usually, it's the amount of milliseconds since January 1, 1970, 12:00:00 AM.

4 hours ago, wasab said:

Is that so. Well, as long as the seed doesn’t repeat, there shouldn’t be a pattern. Mouse position is another way to seed a random although that won’t work if user doesn’t move the mouse at all haha.

 

Mouse position could be an interesting way to seed it!

4 hours ago, tikker said:

 

It's C#, so the .NET API. After a day I'd be more worried about periodicity though, as the RNG's implementation seems to be a little broken. Though that probably doesn't matter for the benchmark, just skew the statistics a bit.

I swapped mine out for XORShift. See this page for more information on that implementation: http://heliosphan.org/fastrandom.html

4 hours ago, DtrollMC said:

I really know nothing about random numbers at all, just when I've pondered it before I recognized System time is a number that's never the same at two moments.

 

4 hours ago, tikker said:

That's true, so it's useful to use as a seed in that way. The problem is that random number generators aren't completely random (hence why they're called pseudo-RNGs). Let's not derail this further though  

Yep! It's more meant to be "sort of" random, but it can't possibly be truly random. See https://www.random.org/ for more information on that.

[TheAmazingDanny]

2 hours ago, M.Yurizaki said:

The thing I'm canting my head about this with regards to using this as a benchmark is what exactly is being exercised here? I'm assuming here, but I'm assuming the gist of the code is basically:

/* Assume some timer event will flip "runBenchmark" to false */
while(runBenchmark){
  headsTailsCounter[rng.next(0,1)]++;
}

Granted maybe the PRNG is heavy in terms of execution, but from a higher level point of view, I'm not sure what this data could be useful for when trying to gauge performance in some other task.

 

It'd be something like:

/* Assume:
- Some timer event will flip "runBenchmark" to false 
- Counter is a 64-bit uint
- Benchmark doesn't ever run long enough for counter to overflow, unless that's the intent.
*/
while(runBenchmark){
  counter ++;
}

And though I understand C# doesn't work like C when it comes to compiling code, I already take issue with either one because I know that it will never exercise the processor that well. Why? Well let's just take the second example. It's a single instruction (it should compile to something like ADD counter 1), but modern processors are capable of instruction level parallelism. Given that Zen and Coffee Lake have four ALUs for ILP, the second task is potentially wasting 75% of how many ALUs there really are in the execution unit. So really, the second benchmark should be:

/* Assume:
- Some timer event will flip "runBenchmark" to false 
- The counters are a 64-bit uint
- Benchmark doesn't ever run long enough for counter to overflow, unless that's the intent.
*/
while(runBenchmark){
  counter1 ++;
  counter2 ++;
  counter3 ++;
  counter4 ++;
}

Of course, this is also making an assumption that the CPU will actually invoke ILP on this.

 

EDIT: On a side note, if you're doing this:

/* Assume some timer event will flip "runBenchmark" to false */
while(runBenchmark){
  int coinSide = rng.next(0,1);
  if (coinSide == 0)
    headsCounter ++;
  else
    tailsCounter ++;
}

Performance may be determined by the branch predictor performance or design, rather than how well the processor can actually perform tasks.

My code is more like:

private void beginTime()
{
    DateTime endTime = DateTime.UtcNow.AddSeconds((double) ExpManager.sharedInstance.endCondition.seconds);
    while (DateTime.UtcNow.CompareTo(endTime) <= 0)
    {
        storeResult(diceRoll());
    }
}

Where diceRoll() generates the random number, and storeResult() increments the counter associated with that roll. I'm not sure if the ALUs have anything to do with threading in this example, but, on my R3 1300X, it uses four threads. If there's more than one ALU per logical thread, then I'm going about this wrong. However, if there's just one ALU per thread, and I'm running this on each thread, then it's hitting all of the ALUs in the processor.

[Mira Yurizaki]

1 hour ago, TheAmazingDanny said:

My code is more like:

private void beginTime()
{
    DateTime endTime = DateTime.UtcNow.AddSeconds((double) ExpManager.sharedInstance.endCondition.seconds);
    while (DateTime.UtcNow.CompareTo(endTime) <= 0)
    {
        storeResult(diceRoll());
    }
}

Where diceRoll() generates the random number, and storeResult() increments the counter associated with that roll. I'm not sure if the ALUs have anything to do with threading in this example, but, on my R3 1300X, it uses four threads. If there's more than one ALU per logical thread, then I'm going about this wrong. However, if there's just one ALU per thread, and I'm running this on each thread, then it's hitting all of the ALUs in the processor.

There are four ALUs on a Zen core and each core can service only two threads, so if you're hitting only one ALU per thread, you're at most utilizing 50% of the total ALU resources. See: (if the picture isn't showing, the link is https://en.wikichip.org/w/images/0/02/zen_block_diagram.svg)

 

However, I'm not sure how the .NET compiles C# code. Though, I'm certain for something simple like an arithmetic operation it shouldn't that out of the ordinary to guess what the ASM looks like.