LTT Labs Test Processes - Documentation

[FlyingApeWithBanana]

Is that all the testing data that is going to get posted or is there going to be more?

 

[LMGcommunity]

Hey all,

 

Here are links to our testing repositories:

Test harnesses used in LTT Labs MarkBench

MarkBench Game Settings

Keras-ocr wrapped in a Python HTTP API for MarkBench tests

[Quadrophenic]

I think it would be really valuable if some of these benchmarks included (and listed) some background software. 

 

I think a huge percentage of users are running Discord and a bunch of chrome tabs and maybe even screen sharing while they're gaming, and some benchmarks that include more of a load like that might be representative. 

[bennieboi20]

16 hours ago, Quadrophenic said:

I think it would be really valuable if some of these benchmarks included (and listed) some background software. 

 

I think a huge percentage of users are running Discord and a bunch of chrome tabs and maybe even screen sharing while they're gaming, and some benchmarks that include more of a load like that might be representative. 

But that’s going to be super subjective. The idea is at base what kind of performance can you expect from xyz cards and processors. Having discord and a “bunch” of chrome tabs is less representative in my opinion.

[Squilleam]

Hello, one small feedback I noticed when looking at the Cyberpunk 2077 benchmark. Why is '-benchmark' flag not being used to be passed to steam, that'll allow you to skip the main menu. Also, CP2077 generates a log file with all the metrics somewhere within it's files after the benchmark scene.

[fulminemizzega]

Making your harnesses public is just amazing. People wasting their time doing tests by hand had to stop a long time ago, this is another great thing done by LMG that will improve how reviews are done for everyone. I hope that other reviewers/organizations also will contribute to the repo instead of just using what you've done.

[theonly500]

One thing i wonder is:

What Average do you use? Is it a arithmetic mean or is it a median?
There is a whole lot to go into there simply due to the complicated mathematics of averages and statistics.

There is a saying, that for one to truly know the data, he needs to know a infinite amount of calculations on that data.
With a bit of math you can encompass things like fluctuation and rate of fluctuation, which can reveal certain things about your test which might not be obvious to a human observer, since we are utter crap at dealing with randomness. And the calulations on performance are based on randomness.

[LTTMobileJake]

On 10/13/2023 at 11:59 AM, theonly500 said:

One thing i wonder is:

What Average do you use? Is it a arithmetic mean or is it a median?
There is a whole lot to go into there simply due to the complicated mathematics of averages and statistics.

There is a saying, that for one to truly know the data, he needs to know a infinite amount of calculations on that data.
With a bit of math you can encompass things like fluctuation and rate of fluctuation, which can reveal certain things about your test which might not be obvious to a human observer, since we are utter crap at dealing with randomness. And the calulations on performance are based on randomness.

PC component testing uses the Arithmetic mean, whereas in Mobile testing (methodologies incoming soon) I tend to prefer Median, but calculate Mean as well

[theonly500]

That is interesting. I would think, that there is some interesting Data in that, especially if you've got your testing methodologies far enough down, that the data sets are so well defined, that you could look into software/hardware limitations and how they impact performance.

There is just so much interesting stuff in the Math behind that data, that it would be awesome to see a investigation into that stuff.

[GoStormPlays]

How would Labs deal with Manufacturers sending you or artificially boosting their products to perform well on some of these tests? I mean, of course some of the tests manufacturers wouldn’t be able to gamify, but I think if some of the devs followed the labs testing processes they might be able to give artificially boosted scores on some of the tests.

[SinusSebastian]

Apologies if this is the wrong thread, didn't see one explicitly for the labs website, but I would be happy to volunteer voiceovers for labs "reviews" based on the script that would other be fed for an AI voiceover, pro bono. And I can't imagine I'd be the only one offering to volunteer.

 

I can send a sample if this idea gains traction. This would be a good way to offer deeper community engagement while exchanging "uncanny valley" territory for actual human feel (at the same price).

 

This is understanding the scope of "all reviews", and while the site is under construction, these can be handled in batches of a couple dozen or so a week (just on my part).