Jump to content

Labs Transparency Thread

LMGcommunity
6 minutes ago, TylerD321 said:

I would like to see the LMG get ISO 9001 certified (or even just the labs) and have the standard work pertaining to video creation or testing methodology public.

THIS would be a solid card to have in their wallet even if it was just labs. I think if the ideology of labs is to challenge the industry on a technical level, they need to be certified AND calibrated like the industry, with openly published records.

Link to comment
Share on other sites

Link to post
Share on other sites

4 minutes ago, qbergeron648 said:

THIS would be a solid card to have in their wallet even if it was just labs. I think if the ideology of labs is to challenge the industry on a technical level, they need to be certified  AND calibrated like the industry, with openly published records.

Wouldn't the engineering tools need to also be calibrated and tested frequently? The Aneco chamber, PSU Tester, Thermal chamber ect? 

 

Link to comment
Share on other sites

Link to post
Share on other sites

8 minutes ago, Toakan said:

Wouldn't the engineering tools need to also be calibrated and tested frequently? The Aneco chamber, PSU Tester, Thermal chamber ect? 

 

Iso is more about process and documenting what you do

Link to comment
Share on other sites

Link to post
Share on other sites

There absolutely needs to be a much, much longer list of hardware with much more extensive graphs with all that hardware in it. It doesn't matter if the text is small and people might need to pause the video... They will if they are making a purchase decision partially based on the data. 

 

But we need at least 3 gens of GPUs included, and 4 gens of CPUs. I want to see 10th gen Intel included with the 14th Gen launch, including i3,i5,i7,i9 in all of them. K and non k. People need a full layout of what they are looking at over time for perspective, not just this Gen vs last Gen with 3 products for each. That's not valuable and doesn't give anyone enough info. Trust your audience is smart enough to deal with the large swath of data. 

Link to comment
Share on other sites

Link to post
Share on other sites

18 minutes ago, Toakan said:

Wouldn't the engineering tools need to also be calibrated and tested frequently? The Aneco chamber, PSU Tester, Thermal chamber ect? 

 

not for ISO 9001, BUT to add to the suggestion that they get ISO 9001, they should also have their machines & equipment regularly load tested/calibrated and publish those cert documents so that it adds to the credibility. 

Link to comment
Share on other sites

Link to post
Share on other sites

2 hours ago, Daniel White said:

Post on GitHub the practices and methodologies in an editable format. Take PRs from the community to improve or add clarity 

Posting it on GitHub or a similar platform that allows diffing with previous versions and seeing exactly what changed and when over time is an excellent idea.
EDIT: with good commit comments ("this change was made to address an issue we ran into when testing motherboard X, where without addition of test Y we were getting incosistent results. The test Y addresses it in the way Z...", etc.) it would actually also be a great demonstration of the work the team is putting into making these methodologies work.

Link to comment
Share on other sites

Link to post
Share on other sites

22 hours ago, LMGcommunity said:

As promised, here is the thread where you can post your transparency suggestions for Labs. Here are some examples of things we'd like to see feedback about:

  • Ideas in our benchmarking.
  • What you'd like to see regarding test system information.
  • Benchmark details.
  • Testbed variations.

And, of course, any other transparency-related feedback is welcome, but similar to the Sponsor Complaints subforum, any off-topic comments will be subject to moderation.

 

Thank you for wanting to help us improve the way we do things here!

Hi,

 

LTT Labs truly excites me, but there are a few challenges that (for the sake of the sustainability of LTT Labs, at least in my opinion) LTT needs to clearly and publicly solve that I'd like to highlight. It's free advice, so take it for what it's worth.

 

Challenge #1: Establish clear guidelines, independent of the product class, for why something should be measured.

Challenge #2: Codify what about that class of thing can/should be measured, and what shouldn't.

Challenge #3: Quantify the one-time (equipment) and recurring (manpower, consumables, opportunity) costs of a given suite of measurements, and clearly label who is bearing that cost.

 

When LTT Labs was announced, Linus positioned it as a swiss army knife to ensure incentive alignment within LTT as a whole, eliminating the need for sponsorships (either fiscal or general rapport) from the companies that sell the products about which LTT makes content. This is wonderfully aspirational, while still being achievable and potentially profitable.

 

Painful thought:

Spoiler

As a viewer, it looks like what happened instead is a shift of content generation responsibility from the writing teams to the Labs... which (if true) would be both dysfunctional and soul crushing for both the writers and technicians in LTT Labs. In my opinion, writers at LTT should not be allowed to dictate needs to LTT Labs in order to make good content. Writers must be empowered/required to have qualitative opinions that are freestanding. A video idea that boils down to a question for Labs to answer or a problem for Labs to solve... doesn't cut it. Writers will be simultaneously too busy mismanaging Labs' projects while appearing unproductive in producing content around Labs' results to have time or desire for creative freedom. 


Excessively high DPI flick shots:

Spoiler
  • LTT Labs must use (whenever possible) measurement standards as ratified by IEEE (or the corresponding professional governing body) in order to have any true credibility. If the standard doesn't exist, you can't just make up your own. You CAN draft a proposed standard (one for each data class for each product class), solicit feedback on the standard here (or at the conventions of the appropriate organizations)... LTX would be a great place for other independent labs and product manufacturers alike to ratify new standards as needed.
  • As has been demonstrated with the keyboard pressing abomination (sic.) project, automation is expensive, has long lead times, and doesn't guarantee repeatability. Anything more precise than is repeatable is a waste of time and resources. On the flip side, an automated test that shows repeatability is indicative of actual data as well as expertise in Labs. Ascribing the repeatable results to the wrong causes WILL happen, but that's not a matter of integrity, just time.
  • Youtube rewards negativity (just look at the hoards of scat fetishism swarming over this current LTT drama), so weaponize that by making a Labs-driven Hall of Shame. Labs gets to pick a product category and quantitative measure, implements the standards, and automates the process. If the measurement isn't weaponized, then the results shouldn't be considered reliable enough to nuke a brand. Sample size of one doesn't count either. Getting manufacturers to continue seeding products will be dependent on Labs position as "the final say" on a statistical basis, with unnecessary bridge-burning parties as a secondary metric.
  • Video series idea: record weekly arguments interactions between writers and Labs breaking down misconceptions and redefining expectations (goes both ways - see literally any audio product for reference). No scripting allowed.
  • If content -> views -> sponsorships/merch sales/ad revenue -> more content, there is no endgame besides siphoning off money at some point. There needs to be a publicly displayed goal for LTT employees and viewers to buy into together, and Labs needs to be a part of that vision. Marketing integrity, better tech products, more open source, more recycling, better documentation, consumer education, just entertainment, etc.?
  • If Labs will ever be anything other than Colton/Dennis-sized expense, there needs to be solid traceability for Lab expenditures. If manufacturers are the funding source, there must be a separation from LTT videos so that manufacturers can expect confidentiality and collaboration. If LTT merch sales are the funding source, there should be a way for viewers to vote with their wallets to shift the focus of Labs to their collective desires (*cough* preorders *cough*). If ad revenue/sponsorships/traditional viewership metrics are the funding source, Labs will fail catastrophically due to public perceptions of conflict of interest. The long lag time between strategic direction provided to Labs and the resulting functional output will guarantee failure if the measure of success is decoupled from the strategic direction. Every company makes mistakes, but nobody can afford to make mistakes that take too long to become obvious.

 

Labs will be a hard vision to execute on, but it's going to be worth it and I am still hopeful that it's possible.

 

Good luck ya'll.

Link to comment
Share on other sites

Link to post
Share on other sites

Would it be possible to semi-regularly run the most up to date testing harness on a known configuration as a canary to detect configuration issues in the testing methodology? I.e. if you knew the expected performance on that known configuration, this canary test would help catch any new issues that may have popped up.

Link to comment
Share on other sites

Link to post
Share on other sites

Videos with LAB’s Data or testing should be explicitly labeled in the thumbnail, Video title and video description as well as on screen when the relevant data is being shown. Also, please publish full lab reports with full data, via PDF with CSV files (for the data) or The Labs website.

Link to comment
Share on other sites

Link to post
Share on other sites

10 hours ago, davejr said:

I found a link to watch the apology video without providing a view on the YouTube video (I cannot believe it was monetized) and saw the statement about how there would be a video describing the process or the details of the flow on Floatplane - I think given what has happened the idea of pay walling the transparency for processes that will provide videos to YouTube shows that no real change of mindset has occurred.

 

If you want to gain the respect and transparency that you allude to, not only should the processes be documented well, but they should be available to anyone who wants them, not just those who pay for FP.

...

I don't want to start debating about this on this thread, however, I do want to mention that as for what I saw, all the documentation of the change was posted in the description of the new LTT video, with Linus mentioning multiple times how it is all available. Making a video takes time, effort, and money, therefore I do expect them to monetize it, along with maybe* talk about the changes in a video on floatplane for easier consumption. But keeping in mind that all data is posted available to the public.  *I don't have access to floatplane therefore I don't know what they posted. As for now I don't see even a locked video with further info.

Link to comment
Share on other sites

Link to post
Share on other sites

This may be obvious but we’ll documented code for any testing software. 
 

you should also define your games by intended type. I.e. E-Sports Title, CPU Heavy Game, GPU heavy game, RTS, FPS, Racing, Sim, old school, optimized, etc. feel free to give them multiple tags. 
 

you should include a link in the description to the raw data for each video. 
 

on a personal level I want Rainbow Six Seige added to the testing suite. 

Link to comment
Share on other sites

Link to post
Share on other sites

Just thought of this but as part of the planning process, the level of effort needs to be defined ahead of time. 
 

for example I/The Community expects for a product launch (New CPU/GPU, etc.) that each test is ran multiple times(3+) and verified by at least 2 different people. 
 

for a video where you are comparing a specific type of part (PSU, Ram, GPU from different manufacturers, etc.) test should be ran multiple times (2+) by the same person at different times/days. This level of effort is meant to weed out any anomalies in the data. 
 

for a low effort video where you just need baseline numbers for a product (pre-built, SBC, handheld, laptop, etc.) a test should be ran multiple times but can be done in secession. 
 

only you guys can define your effort level and you may want to add more levels or less depending on your process. 
 

 

On a completely different note. I have not seen any process defined on how the lab will handle Apple computers. You will not be able to use the same testing suite, so you should generate a separate one. You may want to hire someone to code a game or test suite in a universal language.
 

I would also like to see one for different architectures (ARM, X86, RISC-V,etc.) but that’s definitely a down the line thing. 

Link to comment
Share on other sites

Link to post
Share on other sites

In the interest of transparency, we will need full access to all Labs surveillance cameras. 😛

 

Jokes aside, a designated livecam-equipped station might be cool. (Floatplane exclusive, of course). Obviously would be most interesting for mechanical tests and the like. (GPU-swapping robot, anyone?)

 

Link to comment
Share on other sites

Link to post
Share on other sites

you can like link the graph in your description because people are sometimes not able to view the graphs clearly in the video because the presenter moves on to the next topic. Atleast this can be done until the labs website is made.

Link to comment
Share on other sites

Link to post
Share on other sites

Will you guys consider the new "GPU Busy" metric introduced by Intel with their open source PresentMon software as a new metric to be featured in GPU reviews? Link to the full blog post.

Also a few suggestions :

  • Price , written beside products when displayed on graphs or charts (maybe using msrp or price on various retail sites as of recording ) and compare it with different manufacturers / partners . It would be better for most people to understand large charts especially for those who do not know different naming schemes of various products from same / different manufacturers . 
  •  Visual video capture , of game benchmarks to better understand frame time graphs and frame consistency.
  • Using default settings set by the manufacturer(s), especially for prebuilts / laptop reviews before using optimised settings (like enabling xmp) and showcasing the performance difference as most people only simply use the out of box settings.
Link to comment
Share on other sites

Link to post
Share on other sites

Personally think Labs is a waste of time/money.

 

For example  lets say you review a rtx 5080ti using labs.

It seems like your not going to do a 40+ min deep dive into the card like GN, hardware unboxed and a few other creators do. Nor would I want you to.. GN and others are informative for sure but are very dry and often boring. Whereas your videos are to the point and more light hearted. 

Your data does not have to be industry best. It just has to be accurate.  If some small youtuber can do that with a few thousand subs then you can do it. 

 

I suspect I am of the minority view on this forum when it comes to this though. But I would of much preferred the millions you spent on labs to of gone into creating quality content which is what MrBeast seems to do with a lot of success. 

 

that's my 2cents anyway..
seems like your committed to this labs idea. 

 

Just to go further. 

You have some device that tests keyboards. Does it make your review of a keyboard better for that? Not really. you have more data for sure. But if someone like MKBD tells me a keyboard feels good. I will trust him. Even if your machine notices the E key and A key have slightly different actuation points of a few mm. 

Having loads of data is great and if it informs your review of the product fantastic. But my main takeaway from any video is the reviewers honest opinion of a product not how many different benchmarks people have. 


To be slightly constructive to the labs idea though. 
Maybe try to engage with fellow creators to create a benchmarking platform. It would be pretty awesome for example if all the big players in YouTube submitted their benchmarks and could access each other benchmarks and people could view the average of them benchmarks.
again using a RTX 5080ti as an example it would be cool to see a graph showing  Jayz2cents benchmarks of a  RTX 5080ti with a 13900k compared to you with a 12900k benchmarking the same card. 

 

Would it work realistically? probably not. Too much hassle to get that off the ground and a lot of potential issues with different methodologies and test-benches I suspect.  But it would be cool if it could work because having say 20+ creators submitting benchmarks to one platform is going to get more data then what labs can do. 


Edit:

Oh, I guess a clear distinction between unboxing videos with raw opinions and anything labs based. I suspect a lot of people will jump on any fault in any video even if your just giving a raw view of a product without any serious testing. (Stormbreaker mouse video as an example)

I noticed you posted a pinned message in the comments saying how you deeply regret the error you made.

Which yea, you made a mistake but it was supposed to be an unboxing video with raw first impressions..  

For me anyway.. I see no issue with the Stormbreaker mouse unboxing video. But ideally it should of been clear that this is just an 'initial' impressions / unboxing video. 

I don't want these type of videos to go away due to fear of someone pointing out some flaw.. these types of videos will have flaws because of their very nature. 

 

So yea, please do not ruin your  'raw / unboxing' first impressions videos out of fear of making mistakes. 

Link to comment
Share on other sites

Link to post
Share on other sites

3 hours ago, SureWhyNot777 said:

In the interest of transparency, we will need full access to all Labs surveillance cameras. 😛

 

Jokes aside, a designated livecam-equipped station might be cool. (Floatplane exclusive, of course). Obviously would be most interesting for mechanical tests and the like. (GPU-swapping robot, anyone?)

There are legal reasons why this wouldn't be ideal. First is NDA of anything being tested. The other is employees right to privacy. Third would be whether constant big-brother by hundred anonymous judges improves quality or introduces more stress. Fourth is paywall arguments. Fifth is very propable cuts in the stream and resulting conspiracy theories.

^^^^ That's my post ^^^^
<-- This is me --- That's your scrollbar -->
vvvv Who's there? vvvv

Link to comment
Share on other sites

Link to post
Share on other sites

I would like to see

1) better graphs - steve can do box plots, why not labs? Where it makes sense, it would be great to see run to run variations and how large those actually are

2) ask other specialists in the industry for feedback in each area of testing - maybe even publish how you changed your testing based on that feedback. I am assuming labs people do things to their own standards and it's good to have another set of eyes doing the review of processes. I am not talking about fixing mistakes in a review but instead - what kind of tests are important, how the current ones can be improved, how they should be taken and then visualized for public

3) certifications of labs where it make sense - some ISO standards are full of crap just like anything else, however if a standard is well regarded and used in the industry - use it, maybe educate viewers on why do companies follow it and why it is good to have and go through the certification. Invent your own standards only when there is a good reason to.

 


Documentation - can we get a documentation on:

1) the OS setup that is being used - in the RTX 4060 review it mentions the windows version, patch level, driver version which is all good. But are there any tweaks to the operating system? What about windows updates? Are scheduled tasks controlled? Which apps are running in the background that can skew results? Things of that nature. Same for Linux or Mac.

2) With UEFI - what about SAM/ReBAR - is it enabled? which other settings are controlled or is it all stock as of the specified version? Can we compare the data across reviews?

3) not just how you test each game but also how do you prepare your workbenches

Would be great not just for transparency but also that it would be possible to replicate a specific result

 

Link to comment
Share on other sites

Link to post
Share on other sites

I saw too many GPU review videos with some kind of games like GTA where there are a lot of random components, you can find no traffic in the areas with more graphic detail and a lot of traffic in the less detailed areas and vice versa which gives a randomness to what you see. it happens on the screen and the graph has to do at all times, a single bar that says the FPS only does not say much due to this variation, but in other games like Metro that there is a benchmark in which the same thing always happens the results are more consistent , it only varies minimally in executing them 1 or 100 times.

maybe from time to time made a bigger video with an AMD 3950X and with the same GPUs (maybe something like a 4060, 4070, 3080 and one 4080 using the same GPU at they full potential, and 3 more with different CPU configurations. I say this CPU because is possible used it as a 3950X (2x8) as a 3900X (2x6) as a 3800X (1x8) and as a 3600X (1x6) where the only difference is the number of cores and CCX but everything is the same on that computer... I say the 3950 because it is a beast on it's own and you can disable cores and CCX on the BIOS/EFI on the motherboard to see how the GPU change from one configuration to the next and change only the number of thread of the processor to check the impact the CPU have on that and it can scale with the MHz of the CPU on each model, well some CPU have a different cache configuration but it compare better and can made a baseline of test to compare with. scale the number of MHz with the other CPUs and everything on the system is just the same.

maybe doing something like that will take 16 days (with 1 computer, 8 games and 20 minutes on each resolution, working 78 hours everyday) for each GPU/CPU configuration at 1080p, 2K & 4K.. that don't will made useful information per se but made something to compare to when testing other hardware again as it will scale good enough except when the game will hit a GPU limit or a game limit (fps cap)

I mean, not only made a test for the 4 new GPUs between them and previous test of old card, but also with a this set of "fake" but also real results just to can get a general idea of where the numbers must look like in different scenarios, I think it will help to detect some strange results but at the same time maybe made something than every user can think on when they want to buy a new GPU
-------
another Idea is to don't made only bars on graphs with 0.1% 1% 5% and the average FPS but add the 3 resolutions on the same page with 1080p 2K & 4K so it will made more easy to see the full picture of one card on the same video game and don't need to jump back and forward on the video to see how the numbers change from one resolution to another...

are more types of graphs than only bars, an X/Y plot will be more interesting and easy to see it if it have lines join the points of the graphs than a linear bar to check the difference between resolution and maybe it will show better the picture when we see than a 4060 at 4K isn't a good idea to play CyberPunk 2077 at 4K it barely can run it at 4K@20fps on 2K@60fps at 1080@80fps (this isn't real numbers is just to prove what I try to say) so it will show that the GPU can't work at 4K maybe for the lack or memory or another problem without needing to jump back and forwards on the video to check the differences or hear the audio of the video, I think a lot of the viewers can made their own interpretation of the problem without needing to hear the explanation and then the explanation than is made on the video will be more easy to follow (at least when is only a few GPUs to compare, with a lot of GPUs on the table it will look a bit messy on small screen resolutions)

-------
Another thing to consider is on the powergraph like on that video, maybe scale the timeline will be also a good idea as it will made the difference when the GPU is doing the same task look on the same spot... well it don't will show that the GPU is faster and can do the same task in less time but it will made the graph more easy to read, at the start of that graph is easy to spot the difference, but at the end of it is needed to look ahead in time to check the same spot on the other colour... maybe cut on some parts to don't made the graph continuous but to match this "same tasks" works on the same moment will made it more clear to read , and a few seconds later is just how it look like on the video on the next powergraph
https://youtu.be/3XaOeLPztN4?si=HdSX2gpRdmf5jJbI&t=351

Link to comment
Share on other sites

Link to post
Share on other sites

I would like to see a real processes transparency. You seem to forget that there is a "human" factor that it's more important than automation or configurations: How the hardware is mounted, installed and mantained. I've seen so many errors in hardware handling in videos that I'm pretty sure it's not being correctly taken in consideration.

Link to comment
Share on other sites

Link to post
Share on other sites

personally where NDA and other internal secret's.

id like to see a simple site showing the current test protocol and validation of it using third party but to show an example using Steve a bit old but a good example that you could use as a starting point at least for those data driven people you could even archive each older revision so you can cite them for the older videos' 

 

just a starting point mind you. this is for me a Short and long term LABS suggestion this way when something is fixed or improved the public need simply go to the same link and when the system process undergoes a major revision the site would simple show you the newest by default and then the archive of older data set procedures 

Link to comment
Share on other sites

Link to post
Share on other sites

For gaming benchmarks. Put the game versions in the video, like put it in a corner, marking what's the tested version is (and include the date if possible). Also the tested drivers versions. Plus Windows 11(or 10) versions, especially tested after which patch. (Like now Windows should all get August monthly update patch.)

 

BTW, silicon lottery do matters... Maybe showing like average/highest clock speed? Though maybe only +-50~100Mhz within same CPU/GPU models.

 

I actually think there can't be something messed up during the test, and lab's result. The FPS numbers can't be logged wrong right?

It's all about how will you set the correct BIOS/driver/game settings right in EVERY TEST.

 

For CPUs, really, go watch Steve's video on how to set the "correct" or "what it should be" settings of BIOS settings. Lots of MB manufacturers hide some tweaks (or even different treatment for low/high end MB's BIOS) in CPU turbo behavior, or applying higher than usual voltage, or auto changed RAM's tXX settings (and applying higher than usual voltage).

 

Some slight number deviation, or difference than other reviewers are all OK. It's about getting the end conclusion right.

Link to comment
Share on other sites

Link to post
Share on other sites

Are there plans to improve the methodology for tests other than games? Many previous videos have had critical methodology flaws when it comes to tests, especially those related to linux, machine learning or comparing different ISAs.

 

As one example, you folks mistakenly did a native build of chromium in different ISAs, which is meaningless and has been pointed out twice with no feedback:

On 7/10/2023 at 3:57 PM, igormp said:

The compile test is flawed, both tests should target the same arch (so one of them would be doing a cross-compile). Doing native for each is an entirely different workload.

 

As mentioned already, comparing pricing to regular consumer parts is not ideal. The moment you go for workstation parts the price actually looks cheap.

 

Other than that, yeah, apple's performance claims are ridiculous like they have always been, but that does apply to any other company, so idk what to take from this in the end other that companies are shit and do misleading marketing all the time.

On 7/28/2023 at 11:15 AM, igormp said:

Not their first time doing this, and I doubt it'll be the last. I also voiced this concern in another video of theirs. LTT makes it really hard to give them credibility whenever they try to do anything more serious.

I mean, you can slap asahi on it and should be able to use any GPU in there (even nvidia, since they also offer arm drivers).

 

Another point was in a silly video, where benchmarking really wasn't the point, but still a flaw nonetheless where your results compiling the kernel made absolutely no sense:

On 7/3/2020 at 3:08 PM, igormp said:

Yeah, the test was flawed. Even a 3900x nets sub minute compile times, while a threaripper should be able to do so in less than 30 seconds

 

When it comes to machine learning stuff it seems that you are lacking proper benchmarks, and also don't have a proper methodology, as pointed out bellow by me and other users:

On 2/24/2022 at 7:38 PM, igormp said:

When trying to do such benchmark on windows you're leaving tons of performance on the table, see: https://medium.com/analytics-vidhya/comparing-gpu-performance-for-deep-learning-between-pop-os-ubuntu-and-windows-69aa3973cc1f

 

Even when under WSL2 it's still slower, sadly. Here are my results with a 3060 using fp32 on linux:

image.png.f921e43a89f243082be8a36f2676504f.png

 

And here the results from a 3060ti on windows from an acquaintance:

image.png.d22efa1568a108ec334ac5f2d700ca47.png

 

Whereas the 3060ti should be around 30% faster than my 3060.

 

Anyway, here are some values for comparison that I did some time ago on resnet50:

image.png.545ebd1dc36a07c9a1e02ab266774da1.png

As you can see, the V100 is not that far behind while using a batch size 4x smaller (I didn't try a 256 batch size because there was no point in trying to compare it, I can try it again later), so a higher batch size (2056 maybe?) should be nice to see.

 

That's just the tag price, you can get those way cheaper when you're an actual big company looking to buy many GPUs at once.

 

Sadly you can't just run most of the workloads those nvidia gpus are meant to run. Most of the big ML frameworks are built on top of CUDA, and AMD's software stack is severely lacking when it comes to ML. 

Now if you're looking into some FP64 workloads, then an AMD gpu is what you're looking for (think physics simulations).

 

Sadly LTT is mostly a gamer focused channel, so they're neither really knowledgeable enough to do this kind of stuff, nor would they audience really appreciate it. PugetSystems, ServeTheHome or Level1 on the other hand...

 

I saw that there's a thread for the Labs test processes documentation, but so far it only seems to include games. Would be nice to know if you'll include other kind of tests in there, or if you're going to forgo anything productivity-related at all (which is understandable, given the channel's audience), as long as you make that clear and don't do such flawed tests again without a proper methodology beforehand.

 

 

EDIT: Oh, another nice thing would be to do proper logging of any testbench and release the raw data to the public. Should be easily doable with something like prometheus, and would work for all of your tests in an easy way. Could also do something similar to what Phoronix was been doing for years with http://openbenchmarking.org/

Edited by igormp
add logging suggestion

FX6300 @ 4.2GHz | Gigabyte GA-78LMT-USB3 R2 | Hyper 212x | 3x 8GB + 1x 4GB @ 1600MHz | Gigabyte 2060 Super | Corsair CX650M | LG 43UK6520PSA
ASUS X550LN | i5 4210u | 12GB
Lenovo N23 Yoga

Link to comment
Share on other sites

Link to post
Share on other sites

1 hour ago, Daniel White said:

I posted in another "transparency" thread about using a tool such as GitHub to collaborate on the policies other than on a forum - so we can see what's being change and suggest improvements. I put my words into action this morning. Introducing a GitHub repo to track these policies.

 

https://github.com/daniel-white/lmg-policies

 

I only imported one policy from Colton, but more can be added. I welcome LMG's and the community's input on this experiment.

Link to comment
Share on other sites

Link to post
Share on other sites

Sometimes you guys test CS:GO for some new hardware. It would be cool to also test it with "cybersport" config, like in 1280x1024, Nvidia / amd control panel - best performance for 3d settings etc etc. 

The "benchmark" in CS:GO is relatively useless, you can pull up a demo file from pro matches/high elo pugs and replay them in different settings (at least first 8 rounds). Also declare in the video, which demo you used, which map it was in demo, whom did you spectate from 1st perspective etc.
That would give much more clear understanding of performance, since heavy execute (a lot of grenades being thrown by T side on A/B site) is most demanding and most critical moment in the round (if you are playing the anchor role of A/B site).
sry for bad england

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×