LTT Labs Test Processes - Documentation

[LMGcommunity]

As part of our transparency commitment to the community, we will upload our test procedures for both hardware and software-related items in this thread.  We are starting this process today (See the first reply to this post).

 

Our initial uploads will be in our Bookstack format and then we'll be converting to a version-controlled document for improved auditing and tracking purposes.  We expect to have the vast majority of processes uploaded in September.  And we will continue to provide uploads as we add new test processes and/or modify current processes.

 

Here are links to our testing repositories:

Test harnesses used in LTT Labs MarkBench

MarkBench Game Settings

Keras-ocr wrapped in a Python HTTP API for MarkBench tests

 

We welcome your feedback!

[rexynexus]

Hey folks. Long time lurker, first time poster. I've been the one to primarily work on documentation & data lifespan strategies for Labs, and I've been asked to share some of our internal processes. We've been primarily using an instance of Bookstacks to draft these on, but our plan was to migrate them to a QMS that I've been building in order to better track versions of our processes as they relate to test results we obtain. For the next few weeks, I'll be uploading some screenshots of these Bookstacks documents, and once we've started migrating to that QMS system, I'll be uploading .PDFs instead (which will also contain other things you might expect, like scope, equipment & resource requirements, and version controls). Those .PDFs will be eventually maintained on our website or GitHub as is appropriate, but we wanted to give you folks access to them immediately.

 

Some background on what you're looking at here might also be valuable. When it comes to game tests, we break them out into unique parameter strings in order to remove ambiguity throughout our data collection & analysis process. If you see something like 'Games-1080-Rt', then that means that it's:

1. A game-based test,
2. Conducted at 1920x1080
3. Done using our ray-traced parameter set

We collect all of these parameter sets on a per-game basis. That allows us to revise multiple slim files with changes as needed, instead of having a massive library of all of our game test parameters all in one file that would collect revisions like I collect Steam games. Today I'm releasing our current versions of both Cyberpunk and F1 '23, and I welcome your feedback on it. If you have questions about why we chose one setting over another or what parameter sets we're looking at next for a given game, then this is the thread for you. To start us off, here's a screenshot of the list of game tests we have and are actively working on, the majority of which are harnessed for use with Markbench (or in development for that step). 

[rexynexus]

Cyberpunk 2077 

cyberpunkProcedure_1.png cyberpunkProcedure_2.png

[rexynexus]

F1 '23

[Haaselh0ff]

 

I know the Dota 2 arcade is wild but I had no idea Rocket League was in there lol.

 

Cool to see this stuff!!! Thank you for sharing, love the transparency and openness. Please keep it up!

 

[jos]

I know you guys are planning to do automation and lots of tools are involved. I think you should do something additional. First you need a log of historical data, second you need some option to enter what company promised. Enter your expectation. Then calculate the delta in the variation, this will help to an extent. I know automation is something that will ease things up. But for every benchmark. One random game/ tool etc need to be selected and do a manual testing without any tools, and see how it compares.

[AyesC]

A quick suggestion, you may be able to give MarkBench the ability to set the game settings on its own to avoid human error. A lot of games use generic text-based configuration files to store settings, and are easy to adjust from something like notepad. Almost anything based on Unreal, for example, can easily be adjusted with code, since it stores game settings in a couple of .ini files somewhere in appdata.

[DaiGurenMK42]

Been waiting on this all week. Thanks for sharing.

[SeriousRising]

7 hours ago, rexynexus said:

F1 '23

This isn't really readable in this format please break this up into smaller portions of media

[Kravatie1]

30 minutes ago, SeriousRising said:

This isn't really readable in this format please break this up into smaller portions of media

just open imagine in a new tab

[Guest]

These are software configurations and software applied configurations to the hardware. I need to see the hardware handling processes to avoid, for example, a bad installation of the hardware, bad mounting of a cooler or not enough pressure applied to secure the heathsink or the GPU to the port, etc.

[mpuone404]

9 hours ago, AyesC said:

A quick suggestion, you may be able to give MarkBench the ability to set the game settings on its own to avoid human error. A lot of games use generic text-based configuration files to store settings, and are easy to adjust from something like notepad. Almost anything based on Unreal, for example, can easily be adjusted with code, since it stores game settings in a couple of .ini files somewhere in appdata.

Yeah, I'm surprised they're not doing this already. Having pre made config files that can be automatically injected per game. 

[Toakan]

8 minutes ago, mpuone404 said:

Yeah, I'm surprised they're not doing this already. Having pre made config files that can be automatically injected per game. 

They may already be, Unreal and Unity games can have a few of their common settings (Resolution, configs ect) set through the command line, or even switched to utilise alternate configs this way. 

 

Having those configs shared would be a really good thing, so externally people can download and run them to verify results. People like GN or other networks would also have a way of validating comparisons easily.

Edited August 26, 2023 by Toakan
Had another thought

[Follio]

10 hours ago, AyesC said:

A quick suggestion, you may be able to give MarkBench the ability to set the game settings on its own to avoid human error. A lot of games use generic text-based configuration files to store settings, and are easy to adjust from something like notepad. Almost anything based on Unreal, for example, can easily be adjusted with code, since it stores game settings in a couple of .ini files somewhere in appdata.

 

1 hour ago, mpuone404 said:

Yeah, I'm surprised they're not doing this already. Having pre made config files that can be automatically injected per game. 

 

52 minutes ago, Toakan said:

They may already be, Unreal and Unity games can have a few of their common settings (Resolution, configs ect) set through the command line, or even switched to utilise alternate configs this way. 

 

Having those configs shared would be a really good thing, so externally people can download and run them to verify results. People like GN or other networks would also have a way of validating comparisons easily.

Dev here! Automating the setting of game configuration is a good suggestion, and it's on our roadmap. We aren't currently doing this due to priority; Data collection, analysis, and presentation trump automated game settings for now.

[Toakan]

5 minutes ago, Follio said:

 

 

Dev here! Automating the setting of game configuration is a good suggestion, and it's on our roadmap. We aren't currently doing this due to priority; Data collection, analysis, and presentation trump automated game settings for now.

Surely standardising configuration settings, trumps data collection? 
 

Otherwise the data you collect is subject to run time errors. Setting a standard config, which is locked as read only, stops any data deviation?

[Follio]

Just now, Toakan said:

Surely standardising configuration settings, trumps data collection? 
 

Otherwise the data you collect is subject to run time errors. Setting a standard config, which is locked as read only, stops any data deviation?

Our configuration is standardized, but yes, it is not automated. I agree about the impact of this capability; it's just a matter of when.

[igormp]

17 hours ago, rexynexus said:

We collect all of these parameter sets on a per-game basis.

Is this thread meant just for games? Would be interesting to have something similar to your other productivity stuff.

[Dan Brown]

18 hours ago, rexynexus said:

For the next few weeks, I'll be uploading some screenshots of these Bookstacks documents, and once we've started migrating to that QMS system

Hey @rexynexus, Just signed up here to post after seeing the new LTT video, being a long term viewer of the channel.

 

I'm the creator/maintainer of BookStack.

If easier, you should be able to export at a book/chapter/page level to PDF or contained HTML, which should include the images within (text/markdown export formats won't include images within the content). That said, I'm assuming you might be screenshotting to avoid exposure of internal links/references, and that's totally fair.

 

BookStack does have a REST API which can be used for export and management of content, in a few different formats. I have a bunch of existing scripts in a repo here. You could use these to automate export (and maybe auto-remove links/references as part of that) for the current sharing you're doing, and/or use this to help that future migration.

 

If you need any help/pointers on this feel free to reach out to me.

[rexynexus]

3 hours ago, Dan Brown said:

Hey @rexynexus, Just signed up here to post after seeing the new LTT video, being a long term viewer of the channel.

 

I'm the creator/maintainer of BookStack.

If easier, you should be able to export at a book/chapter/page level to PDF or contained HTML, which should include the images within (text/markdown export formats won't include images within the content). That said, I'm assuming you might be screenshotting to avoid exposure of internal links/references, and that's totally fair.

 

BookStack does have a REST API which can be used for export and management of content, in a few different formats. I have a bunch of existing scripts in a repo here. You could use these to automate export (and maybe auto-remove links/references as part of that) for the current sharing you're doing, and/or use this to help that future migration.

 

If you need any help/pointers on this feel free to reach out to me.

Hey Dan! Thanks for the comment. You’re correct on the link considerations - we should be migrated soon enough that I’ll just keep screenshotting rather than exporting until we can get out properly version-controlled .PDF files, but I appreciate the suggestions. BookStack has been great for drafting things on the fly with its easy revision systems, by the way! If I wasn’t already integrated into Obsidian + HedgeDocs on my homelab, I’d probably look into using it myself at home. 

[Mr.man1093]

I noticed that in the screenshots for both Cyberpunk and F1 32 that Motion blur was turned on. I'd recommend disabling motion blur on all benchmarked games as most people are going to play with it off. Testing data recorded with motion blur on won't be indicative of the real-world performance that most people will see.

[Dellyjoughnut]

Will there be inclusion and exclusion criteria included in the documentation released to the public for each experiment? 

[Sapphiree]

What about test locations in games for cpu/gpu load scenarios respectively? Built-in benchmarks are often subject to driver optimization by manufacturers and can distort the whole picture.

[Cipriantheking14]

Hi! I was wondering if you guys are also going to release the testing procedure for mobile games/benchmarks if it will happen (please release them tho)?

[FlyingApeWithBanana]

At the bottom of the F1 '23 screenshot it says that the results should be uploaded to the uploads tab are we going to get screenshots of that as well?

[LTTMobileJake]

On 8/30/2023 at 3:37 PM, Cipriantheking14 said:

Hi! I was wondering if you guys are also going to release the testing procedure for mobile games/benchmarks if it will happen (please release them tho)?

Yes we will be!

On 9/6/2023 at 8:42 PM, FlyingApeWithBanana said:

At the bottom of the F1 '23 screenshot it says that the results should be uploaded to the uploads tab are we going to get screenshots of that as well?

This is referencing the MarkBench interface's upload tab - it just lists the sessions with an upload button and input for server IP