Labs PSU Testing Credibility / Conflict of Interest

[yolosnail]

I made some comments on Floatplane questioning the credibility of the PSU testing given that Seasonic helped them spec out the testing equipment and sponsored a video unboxing it, and unsurprisingly my comment was downvoted into oblivion. 

 

To me, it's no different from Labs wanting to build a CPU benchmark, so they ask Intel for help in designing said benchmark. Intel then sponsor an LTT video showing off the new benchmark. 

 

LTT then frame the sponsorship as Intel having big ballz because the benchmark has the potential to make their CPU look bad compared to the competition, despite Intel actually helping with the design of the benchmark.

 

Would you not be a bit suspicious of the reviews formed from the results of the benchmark?

 

I'm not saying Seasonic have done anything nefarious (and they probably haven't), but how do we know that they haven't lead Labs down a path with the equipment that favours their power supplies over another manufacturer?

 

The fact that they seem to be basically just doing a Ctrl+C and Ctrl+V of Seasonic's testing means that Seasonic will presumably do very well in the tests, whereas another manufacturer, which tests things slightly differently, may not. I feel like this is something that needs to be addressed in more detail, especially if they're basing their testing methodology on Seasonic's

 

So, am I alone in having those concerns?

[ToboRobot]

3 minutes ago, yolosnail said:

I made some comments on Floatplane questioning the credibility of the PSU testing given that Seasonic helped them spec out the testing equipment and sponsored a video unboxing it, and unsurprisingly my comment was downvoted into oblivion. 

 

To me, it's no different from Labs wanting to build a CPU benchmark, so they ask Intel for help in designing said benchmark. Intel then sponsor an LTT video showing off the new benchmark. 

 

LTT then frame the sponsorship as Intel having big ballz because the benchmark has the potential to make their CPU look bad compared to the competition, despite Intel actually helping with the design of the benchmark.

 

Would you not be a bit suspicious of the reviews formed from the results of the benchmark?

 

I'm not saying Seasonic have done anything nefarious (and they probably haven't), but how do we know that they haven't lead Labs down a path with the equipment that favours their power supplies over another manufacturer?

 

So, am I alone in having those concerns?

It sounds like Seasonic helped them buy a specialized tool, not that Seasonic is going to design or limit their testing process in any way other than the acquisition of the tool.

Given that they haven't even done any testing yet, your accusations are based entirely on speculation.  Do we know if they also consulted with people at Corsair, Gamers Nexus, Igors Labs as well?  Do we have any abnormal test results?  Have they published any results that don't match those of other testers?

 

[yolosnail]

1 minute ago, ToboRobot said:

It sounds like Seasonic helped them buy a specialized tool, not that Seasonic is going to design or limit their testing process in any way other than the acquisition of the tool.

Given that they haven't even done any testing yet, your accusations are based entirely on speculation.  Do we know if they also consulted with people at Corsair, Gamers Nexus, Igors Labs as well?  Do we have any abnormal test results?  Have they published any results that don't match those of other testers?

 

This is why I feel like they need to go address how much input Seasonic had into the testing.

If their input is just "this machine good", then fair enough.

[Kopaka]

It crossed my mind too, but Kyle also said he spent six months researching it, so I'm sure he's had time to make sure it's not biased as far as possible. Just getting a copy of seasonic's wouldn't take that long.

[Nystemy]

As an electrical engineer myself I frankly don't see any room for Seasonic to skew the test setup in their favor.

 

The tests the setup is meant to perform are industry standard tests done to any PSU put onto the market.

 

This being things like:

• Load testing both static and transient.
• Mains noise immunity testing. 
• PSU efficiency.
• Power factor correction efficacy.
• Output noise and ripple.
• Testing the response of protection features like over current, over power, over voltage and short circuit.
• And repeating the above at different input voltages and frequencies.

This is standard stuff, I have done this to a fair slew of designs I have made over the years myself.

 

Next thing in the electrical engineering field is that these tests generate objective data.

 

Transient response for an example is measured in Watts per second, or Amps per second, or volts per second _{(depending on the type of transient)}. There is no nuance behind that number, it is the fastest change in load the PSU can cope with, nothing more or less. _{(% per second is also used in the industry and is sometimes more useful in practice. This depends on the application but effectively it is the same thing but we convert it to a percentage of peak output. Also, the actual transient response will vary in practice, since it is typically a curve relative to current load.)}

 

Same story for output ripple, it is simply how many volts the output voltage varies with over a relatively short period of time. _{(While noise is over a much shorter period of time. Ripple is typically anything bellow about 1-10 kHz in the PSU industry, while everything up to 20 MHz is considered noise. And this is based on an industry standard that is used for all power supplies, not just computer/server ones, but everything on the whole grid and beyond as far as mains power is concerned. (for higher frequency applications voltage changes up in the many GHz can be considered ripple at times. Last I checked Intel talked about a few hundred MHz as "ripple" as far as their CPUs are concerned. But the main difference between "ripple" and "noise" is that "ripple" makes the thing not work if the ripple is too large, while "noise" is just an annoyance if there is too much of it. Oversimplification and it greatly depends on the product being powered.))}

 

Power Factor Correction is likewise governed by a standard. _{(One compares the real power consumed to the apparent power consumed, the ratio between these is 1 if the apparent and real power is qual. If the real power is smaller then the ratio goes bellow 1, if the apparent power is smaller then the real then one broke physics.)}

 

Similar story for testing protection features. Here one just needs to measure how far above baseline values on needed to go before the feature kicked in, and how long it took for it to kick in. And the baseline values is what the manufacturer claims as the products specs. _{(And sometimes the manufacturer even gives rating for the protection features like "OCP at 125%", then one knows that at 125% of rated current it should trip, then we can measure how far off that were in reality.)}

 

 

There is plenty of things one can worry about, but in the field of power supplies things are too objective to leave room for skewing tests in favor of a specific manufacturer.

 

Unlike CPUs, there isn't a multitude of nuances, a Volt is a volt regardless of what manufacturer made the PSU, same for an Amp, Second, or Hz. And electronics test equipment is likewise built to adhere as tightly as economically possible to internationally recognized standards. There is some variance between test setups, but generally speaking it is much smaller than the unit to unit variance of PSUs. _{(when I say much smaller, I am not talking a few percent here, but rather an order of magnitude better accuracy is the industry standard to saying that you measured a value to a given resolution.)}

 

In the end.

Don't worry about the trustworthiness of this PSU test setup, it is boringly bog standard to people in the industry...

[OrdinaryPhil]

I do not see how SeaSonic sponsoring the video is a conflict of interest. It sounded to me like they acted in an advisory capacity when the Labs team was selecting the proper tools for testing - testing that (if performed correctly) cannot be biased towards SeaSonic.

[Gaires]

15 hours ago, OrdinaryPhil said:

I do not see how SeaSonic sponsoring the video is a conflict of interest.

The problem is human one. Since Seasonic has been heavily involved in the process, to the point where they sponsored a video, Kyle might be influenced to give Seasonic some leeway when it comes to the results. It might also go otherway around if Kyle feel pressured by the conflict of interest and thus be too harsh on Seasonic to "prove there isn't a conflict of interest".

 

Also people need to remember that conflict of interest doesn't mean that there is one, it just mean there can be, which is why it's so problematic.

[GDRRiley]

seasonic was not the only people they worked with and its standard industry hardware
I'd be more suspicious if they worked on the testing order/plan

[LogicalDrm]

8 hours ago, Gaires said:

The problem is human one. Since Seasonic has been heavily involved in the process, to the point where they sponsored a video, Kyle might be influenced to give Seasonic some leeway when it comes to the results. It might also go otherway around if Kyle feel pressured by the conflict of interest and thus be too harsh on Seasonic to "prove there isn't a conflict of interest".

 

Also people need to remember that conflict of interest doesn't mean that there is one, it just mean there can be, which is why it's so problematic.

So couple of things to note. First is, LMG is heavily supported by sponsors, a lot of them. And a lot of them PSU manufacturers. Would Seasonic be favored over Corsair because they only sponsor PSU content while Corsair sponsors other stuff too? Doubtful. The good thing about getting money from everyone is that you are treating all of them equally. Be that to good or bad.

 

Second major thing is cross-checking. Lab isn't, won't be, doesn't try to be only PSU reviewing and testing unit. Any results they will provide can be argued by another reviewer. If the Labs will public white paper style content, they will have detailed information on how testing has been done so the results can be replicated, with margin of error noted. Thats how scientific research works and why its called "peer-reviewed". So would them praising Seasonic over the competition benefit them? Nope, it would wreck the credibility on that one instance someone called them out on any tricks.

 

Third, I don't think calling this kind of thing "conflict of interest" is the proper way to use that term. I'm not sure what the correct term is, but seems bit blowing things out of portion to me. Besides, LMGs Labs won't be first nor last reviewing site to get freebies and ad/sponsor money. Thats how all review sites are run.

 

Lastly, about FP comments. Thats place where HC Linus fans are. They can't stand ANY criticism, even something made in joking manner. I once noted that @CPotter drives huge SUV and how thats "very tree friendly and environmental to show" just after Mr.Beast tree planting collab. They didn't like that and thought I was bullying... So I wouldn't mind downvotes there. LMG will still read the comments.

[LAwLz]

Haven't watched the video, but my opinion is that a manufacturer should not be involved in any way shape or from when building and designing tests for their products.

 

Imagine if they asked Intel which programs they would use to benchmark CPUs, or if Nvidia paid LTT to use certain games when they would compare AMD vs Nvidia GPUs.

 

Even IF the advice is solid, Intel gave them really good benchmarks and Nvidia just gave them a list of popular games, there is still very much room for the sponsor to influence the outcomes of the tests.

Even if they didn't end up influencing the outcome of the tests, it is still very bad optics.

How often do serious scientific reviews get dismissed because they were funded by a company that has vested interests? Would you trust a research paper on how soda consumption affects peoples' health if it was funded and conducted by Coca Cola? Would you trust a study from Sony and Nintendo on the health benefits of playing video games? Even IF the studies are accurate, the fact that they are funded by companies with conflicting interests puts doubt on the studies. 

 

 

I didn't have high hopes for LTT Labs when it was announced, my trust for it didn't exactly go up when it was announced it would be lead by a marketing director, and hearing that for example their PSU testing facility is sponsored by Seasonic does not really put my worries to ease. If anything, every announcement surrounding the lab has been the exact opposite of what I was hoping for. And that's a shame because I really want the lab to succeed. We need more outlets with proper testing.

[Nystemy]

1 hour ago, LAwLz said:

their PSU testing facility is sponsored by Seasonic

That were however not stated in the video.

What were stated were that the sponsor of the video were Seasonic.

 

Not the sponsor of the equipment.

And not the sponsor of future running costs of said equipment.

 

Just the sponsor for the video talking about the equipment. _{(since almost every LTT video has a sponsor spot in it. Effectively an ad space for the highest bidder that LMG is morally willing to work with.)}

 

Don't make a hen out of a feather.

But yes, the sponsor money will cover some of the equipment and running costs. _{(likely not covering much at all, since sponsor spots aren't actually all that expensive given the statements made in the "how we make money" videos.)} So even monetarily there isn't much conflict of interest, and it isn't like Corsair also sponsors a fair amount of videos, Gigabyte too, among other PSU manufacturers.

 

And there is plenty of standards for PSU testing, and if Lab32 releases pure measured values then there won't be room for a skewed opinion. Secondly it would damage the reputation of the lab as well long term, so there is more to loose from it than the little money potentially gained. Even from Seasonic's own standpoint.

 

But in short, our moderator here sums it up nicely:

1 hour ago, LogicalDrm said:

I don't think calling this kind of thing "conflict of interest" is the proper way to use that term. I'm not sure what the correct term is, but seems bit blowing things out of portion to me. Besides, LMGs Labs won't be first nor last reviewing site to get freebies and ad/sponsor money. Thats how all review sites are run.

 

And then we have the most important part a lot of LTT viewers seems to completely forget....

Lab32 don't have to compete out the other reviewers doing more lab based testing, nor do they have to compete with actual test labs. _{(Nor won't they be able to compete with accredited test labs.)}

This isn't just the case for reviews, but also for a lot of actual production testing and certifications testing too. _{(As an example, even the Underwriters Laboratory, an FCC accredited test lab still does goof once in a blue moon when it comes to very basic tests, having a second or even third source for validation is critical if one wants to get the product to market in a timely fashion, since else one can't call out the UL results as screwy and have them retest the product again for "free" (still costs you plenty of time! Time is sometimes more expensive than actual money.). The more lenient EU regulations sometimes allows for in house testing and proper documentation and why some products gets released in the EU months before they hit the US market. However, most times the second/third source is often a non accredited test lab that does the same tests to give one an idea if one will pass the accredited test or not, this is particularly useful for development since non accredited tests are way cheaper, and can often also be done in house.)}

[Gaires]

9 hours ago, LogicalDrm said:

First is, LMG is heavily supported by sponsors, a lot of them.

Which is a conflict of interest if they also review those sponsor's prducts, this also includes the free samples.

 

9 hours ago, LogicalDrm said:

The good thing about getting money from everyone is that you are treating all of them equally.

Not really, some sponsors give you more money compared to the others. They might also give you other preferential treatment.

 

9 hours ago, LogicalDrm said:

Lab isn't, won't be, doesn't try to be only PSU reviewing and testing unit. Any results they will provide can be argued by another reviewer.

Problem here is that there aren't many PSU reviewers, since the hardware is too expensive and there aren't many people reading those reviews meaning it's really risky endeavor. You need to spend a lot of money upfront and then hope people will come and read your reviews, which are hard to understand for most people.

So say the Lab gets slightly different results compared to some other reviewers, they don't have the time to go and verify the Lab's results.

 

9 hours ago, LogicalDrm said:

So would them praising Seasonic over the competition benefit them? Nope, it would wreck the credibility on that one instance someone called them out on any tricks.

It doesn't mean they have to praise Seasonic to the high heaven and back, it just means they might understate some problems on Seasonic units.

 

9 hours ago, LogicalDrm said:

Third, I don't think calling this kind of thing "conflict of interest" is the proper way to use that term.

From Google "conflict of interest: a situation in which the concerns or aims of two different parties are incompatible.".

So when it comes to manufacturer - reviewer relationship the problem is that the manufacturer wants positive coverage and the reviewer is supposed to be neutral but is dependant on the manufacturer.

 

9 hours ago, LogicalDrm said:

LMGs Labs won't be first nor last reviewing site to get freebies and ad/sponsor money. Thats how all review sites are run

Sure, and it's a conflict of interest. You can look at the Hardware Unboxed situation where Nvidia felt they had enough leverage to force them to review their products in more positive light. They felt they can bully one smaller reviewer into doing their bidding, because even if they lost HU coverage they would still have many more reviewers to review their products.

You can also look at the gaming journalism where the reviewers are dependant on the game studios to give them press copies, and thus at least some sites do blatantly give those games good scores.

 

And I again emphasize that this doesn't mean that there 100% is a conflict of interest, it just means there can be one. Which is problematic since we as a consumers will most likely never know if there is or isn't one.

[Akima]

On 8/24/2022 at 8:34 PM, Nystemy said:

As an electrical engineer myself I frankly don't see any room for Seasonic to skew the test setup in their favor.

 

The tests the setup is meant to perform are industry standard tests done to any PSU put onto the market.

 

This being things like:

• Load testing both static and transient.
• Mains noise immunity testing. 
• PSU efficiency.
• Power factor correction efficacy.
• Output noise and ripple.
• Testing the response of protection features like over current, over power, over voltage and short circuit.
• And repeating the above at different input voltages and frequencies.

This is standard stuff, I have done this to a fair slew of designs I have made over the years myself.

 

Billion percent this. When it comes to EMC and compatiability testing - everything is driven by gobally defined standards - IEC/BS EN/ISO, etc etc.

 

For equipment to be sold as a means of testing these standards, they have to meet the requirements. So it's not like Seasonic can suggest using a particular brand of testing equipment because it gives them favorable results. They're either going to pass the standards, or they're not. 

[LogicalDrm]

2 hours ago, Gaires said:

Not really, some sponsors give you more money compared to the others. They might also give you other preferential treatment.

Well, as the sponsor contracts aren't public, we only have what Linus has said to go with. And what has said in public states that majority of the contracts made have fixed rate. This was brought up with the controversy around Raid: Shadowlegends sponsorship where community argument was that they were offered more by Raid than some others.

[Sarra]

Getting assistance from a sponsor on the specs of testing equipment is NOT a conflict of interest.

 

Getting assistance to buy testing equipment from a sponsor is NOT a conflict of interest (depending on how far removed the sponsor is from the actual purchase).

 

Buying the testing equipment FROM the sponsor IS a conflict of interest.

 

LMG ticks the first box, and potentially ticks the second, but not the third.

 

I doubt that Seasonic makes testing equipment of the scale that LMG purchased, I'm sure they have specialized in-house only testing equipment that they design and build, but it's just that... Specialized.

 

I think that, in the future, keeping in mind where the equipment was specced out and how it was purchased is important, but I don't think it can be used to discount the validity of the test results.

 

The only potential future conflict I can see is if a sponsor provides technical assistance in setting testing procedures. This might be something that LMG could do a video on, to demonstrate that they are truly independent in this sense.

[Ravendarat]

I swear the "community" as of late exists only to make a mountain out of a mole hill. The video was sponsored by Seasonic, and that was disclosed. Does that matter, nope. Is it a conflict of interest, of course not. Is them talking to Seasonic about what kind of testing equipment to get a conflict of interest? Nope, its smart research. Seasonic is a forefront in the industry in regards to power supplies, why wouldnt you want input from some one like that. They arent making the equipment so they can weigh the machine to work better with their stuff, they are simply educating. Labs did their own research and chose their equipment layout for what they felt was needed. At some point some of you people are just gonna have to start trusting others until they give you a VALID reason not to. This whole "question everything/ everyone" attitude that is being abused lately is annoying and producing a lot of unneeded white noise. If you are sure that there is no credibility left in the world than go hide in a bunker and wait for the end because "the sky is falling" attitude is played out. I swear some people think that kind of thinking makes them smart, but you come off as the opposite.