What do you think about LTT Lab being used for channels like ShortCircuit?

[e22big]

So I've watched SC video about Pixel 7a yesterday and notice something strange. They've asked the Lab to test the phone for them and the result shows that the 7a has nearly 50 percent better battery life than the regular 7 which they emphasised a lot.

 

But then I've watched Dev video about the it and he shows that they have pretty much the same battery life, under the same testing methodology and both screen calibrated to the same 60hz. Which is actually reasonable, they both have the same battery size, probably the same screen, and the same chip, running the same software, how can one of them run so significantly more efficient than the other?

 

That's when I've realised that SC data from the Lab is probably invalid, the 7a shipped with 60hz out of the box while the regular 7 came with 90hz and I've got the feeling that they probably just test it out of the box like that and never bother to calibrate the screen to the same refresh rate (they've never disclose their testing methodology in details) - and it's probably what result in a huge disparity of a battery life between these two devices. 

 

This maybe a fun, unbox type channel, but if they are publishing data, I think they needed to be able to validate it otherwise it can be highly misleading to the viewers (as with the case here, if true, at least I wished they published a methodology for their test.) If they don't want their video to be taken seriously, I would rather they never release any of the testing data to begin with. There should be a clear separation between an unboxed video and a review video imo, a data should only be published when you review a product, with all of the due diligence required for a proper review. 

 

Am I just mad? What's your thought on this?

[matt0725]

While i think they should disclose testing methods for comparison with others and for our own knowledge, the overwhelming majority of consumers never open their settings app other than for blueooth and wifi, so testing the phone as it will be for most consumers is not a bad idea in itself.

 

Now did they plan for this due to that logic or was it a mistake? Only they know until the labs website which will have testing info included goes live.

[OhioYJ]

15 minutes ago, e22big said:

That's when I've realised that SC data from the Lab is probably invalid, the 7a shipped with 60hz out of the box while the regular 7 came with 90hz and I've got the feeling that they probably just test it out of the box like that and never bother to calibrate the screen to the same refresh rate (they've never disclose their testing methodology in details) - and it's probably what result in a huge disparity of a battery life between these two devices.

Why wouldn't they test it like it came out of the box? I can under volt and under clock my phones CPU / GPU as well? I can modify my display settings to further dim the display.  Where do you draw the line? I think running it how it came is the most realistic test honestly, as that's how most users would run the phone. Well actually most users are going to install 100 different apps, then wonder why their battery life is worse than claimed / stated.

[e22big]

51 minutes ago, OhioYJ said:

Why wouldn't they test it like it came out of the box? I can under volt and under clock my phones CPU / GPU as well? I can modify my display settings to further dim the display.  Where do you draw the line? I think running it how it came is the most realistic test honestly, as that's how most users would run the phone. Well actually most users are going to install 100 different apps, then wonder why their battery life is worse than claimed / stated.

 

Not at 60hz, and high refresh rate display isn't cheap and this is a very expensive midrange phone, if you bought a 90hz phone I am very sure you will want it run at 90hz (it's even cause a huge controversy back in the day with Pixel 4.) If you don't care about 90hz, you would have bought the 6a instead of 7a, that was one of the biggest factor that set the two apart.

 

And it's not like your CPU OV or UV where you need to pretty much hack the processor to perform the way you like, display option is just right there and I am pretty sure most people tangle with it (for something like, getting the 1440p output when most shipped with 1080p out of the box)

 

But let's suppose you're right and most people only care about the battery behaviour right out of the box, even willing to drop one of the major features they've paid for because they can't spend 10 second, it's still not ok to just conclude that 'this phone has a major battery advantage' without all of the asterisk fully disclosed. If you tested one phone at 90hz and another at 60hz because that's how they shipped right out of the box, you need to let the people know - because that's kind of important for the people who care too. 

 

And if you can't be bothered to do that, then it's better to like I've said, just don't publish the data. At least you don't spread the misinformation around.

[johnno23]

I am of the opinion test it as is out of the box.

That's how it is marketed and sold to the consumer.

If the consumer wishes to adjust things that is their choice so I do not see it as misinformation.

you read the adverts you read what is on the box if it behaves different to what you expect then there is an issue.

The words from Linus at the outset were to test products against what the vendors advertised to ensure people would get what they actually paid for.

So dropping a phone screen refresh rate to match a different product even from the same vendor would be wrong in this scenario.

 

[crazzp]

4 hours ago, e22big said:

That's when I've realised that SC data from the Lab is probably invalid, the 7a shipped with 60hz out of the box while the regular 7 came with 90hz and I've got the feeling that they probably just test it out of the box like that and never bother to calibrate the screen to the same refresh rate (they've never disclose their testing methodology in details) - and it's probably what result in a huge disparity of a battery life between these two devices. 

Many tests have shown that 60 and 90hz will not product a 50% difference in battery. At most it's a 10-15% difference. 

 

3 hours ago, e22big said:

But let's suppose you're right and most people only care about the battery behaviour right out of the box, even willing to drop one of the major features they've paid for because they can't spend 10 second, it's still not ok to just conclude that 'this phone has a major battery advantage' without all of the asterisk fully disclosed. If you tested one phone at 90hz and another at 60hz because that's how they shipped right out of the box, you need to let the people know - because that's kind of important for the people who care too. 

 

And if you can't be bothered to do that, then it's better to like I've said, just don't publish the data. At least you don't spread the misinformation around.

You're just speculating at this point. You have no evidence that that they test it at 60hz. Though it's pretty funny you assume it so even when you know how technical the labs people are. 
 

4 hours ago, e22big said:

a data should only be published when you review a product, with all of the due diligence required for a proper review. 

No. Data can be published anytime they want. Due diligence should be required all the time. 

 

Why are people holding LTT to some superior being that needs to have unquestionable morality. They are a media company, not scientific journals or academia. They try their best, but listen to their opinion with a pinch of salt and skepticism, all the time. 

[OhioYJ]

11 hours ago, e22big said:

 

Not at 60hz, and high refresh rate display isn't cheap and this is a very expensive midrange phone, if you bought a 90hz phone I am very sure you will want it run at 90hz (it's even cause a huge controversy back in the day with Pixel 4.) If you don't care about 90hz, you would have bought the 6a instead of 7a, that was one of the biggest factor that set the two apart.

 

I'm replying from a phone with a 120 Hz screen that has been set on 60 Hz since day one. 

[CeePeeBee]

12 hours ago, johnno23 said:

I am of the opinion test it as is out of the box.

That's how it is marketed and sold to the consumer.

<snip>

 

 

Here's my take. If you're going to test a product, it's because you are evaluating it against the competition. That wasn't the original plan for SC as I remember it. SC was a return to the old-style unboxing, where the product itself was judged on its own merit.

 

Taking into consideration the quote above, that's like saying you shouldn't take into consideration how good AC is on a car because it comes with it turned off. If you're going to evaluate, then you need to evaluate the features that are sold as is to consumers. But that's where we get into "review" territory for me.

 

I just wanna see somebody unbox a product, and say how it works and what they like or don't like about it. If I want to gauge whether it's a good product or not in the wider market, I'll check out a review.

[johnno23]

thankyou for taking something out of context to manipulate as you feel fit.

if I buy a car that has AC on the list of selling points I expect to be able to turn it on or off.

if the car was sold as having AC but it had none then it would be false advertising.

Reviews are fine but I refer to the simple facts that when a product is advertised to me as a consumer that can can do specific things but in reality fails to comply then it is good that these items on the list are fact checked. it is simply that simple.

 

[e22big]

On 5/12/2023 at 10:54 AM, crazzp said:

Many tests have shown that 60 and 90hz will not product a 50% difference in battery. At most it's a 10-15% difference. 

 

You're just speculating at this point. You have no evidence that that they test it at 60hz. Though it's pretty funny you assume it so even when you know how technical the labs people are. 
 

No. Data can be published anytime they want. Due diligence should be required all the time. 

 

Why are people holding LTT to some superior being that needs to have unquestionable morality. They are a media company, not scientific journals or academia. They try their best, but listen to their opinion with a pinch of salt and skepticism, all the time. 

 

True it's still all a speculation at this point, but it's still a fact that the data looks incredibly strange, considering the context. And we would never have to have this conversation had they not publish it without methodology, or made a conclusion base on it to begin with. Which arguably is not even the original intent of SC

 

LTT may not be a scientific community but they usually are more thorough than this, and if you compare the Lab to a site like rting where they tried to compete with - rting will a 100 percent respond to an inquiry if you notice some irregularity in their methodology and result, they have staffs dedicated for it and they actually get to the bottom of every of their concern. SC thus far, had never shown this level of dedication - which as I've said, that's fine. 

 

Just don't publish the data and pretend to be something you aren't. There's a limit to how deep you could go as a first impression channel vs a review channel