LTT Labs Test Processes - Documentation

[LMGcommunity]

As part of our transparency commitment to the community, we will upload our test procedures for both hardware and software-related items in this thread.  We are starting this process today (See the first reply to this post).

 

Our initial uploads will be in our Bookstack format and then we'll be converting to a version-controlled document for improved auditing and tracking purposes.  We expect to have the vast majority of processes uploaded in September.  And we will continue to provide uploads as we add new test processes and/or modify current processes.

 

We welcome your feedback!

[rexynexus]

Hey folks. Long time lurker, first time poster. I've been the one to primarily work on documentation & data lifespan strategies for Labs, and I've been asked to share some of our internal processes. We've been primarily using an instance of Bookstacks to draft these on, but our plan was to migrate them to a QMS that I've been building in order to better track versions of our processes as they relate to test results we obtain. For the next few weeks, I'll be uploading some screenshots of these Bookstacks documents, and once we've started migrating to that QMS system, I'll be uploading .PDFs instead (which will also contain other things you might expect, like scope, equipment & resource requirements, and version controls). Those .PDFs will be eventually maintained on our website or GitHub as is appropriate, but we wanted to give you folks access to them immediately.

 

Some background on what you're looking at here might also be valuable. When it comes to game tests, we break them out into unique parameter strings in order to remove ambiguity throughout our data collection & analysis process. If you see something like 'Games-1080-Rt', then that means that it's:

1. A game-based test,
2. Conducted at 1920x1080
3. Done using our ray-traced parameter set

We collect all of these parameter sets on a per-game basis. That allows us to revise multiple slim files with changes as needed, instead of having a massive library of all of our game test parameters all in one file that would collect revisions like I collect Steam games. Today I'm releasing our current versions of both Cyberpunk and F1 '23, and I welcome your feedback on it. If you have questions about why we chose one setting over another or what parameter sets we're looking at next for a given game, then this is the thread for you. To start us off, here's a screenshot of the list of game tests we have and are actively working on, the majority of which are harnessed for use with Markbench (or in development for that step). 

[rexynexus]

Cyberpunk 2077 

cyberpunkProcedure_1.png cyberpunkProcedure_2.png

[rexynexus]

F1 '23

[Haaselh0ff]

 

I know the Dota 2 arcade is wild but I had no idea Rocket League was in there lol.

 

Cool to see this stuff!!! Thank you for sharing, love the transparency and openness. Please keep it up!

 

[jos]

I know you guys are planning to do automation and lots of tools are involved. I think you should do something additional. First you need a log of historical data, second you need some option to enter what company promised. Enter your expectation. Then calculate the delta in the variation, this will help to an extent. I know automation is something that will ease things up. But for every benchmark. One random game/ tool etc need to be selected and do a manual testing without any tools, and see how it compares.

[AyesC]

A quick suggestion, you may be able to give MarkBench the ability to set the game settings on its own to avoid human error. A lot of games use generic text-based configuration files to store settings, and are easy to adjust from something like notepad. Almost anything based on Unreal, for example, can easily be adjusted with code, since it stores game settings in a couple of .ini files somewhere in appdata.

[DaiGurenMK42]

Been waiting on this all week. Thanks for sharing.