LTT Video Error Handling Standard Operating Procedure (SOP)

[Uttamattamakin]

Looking at the "here's the plan video".  This all sounds like the things I said they ought to being done.  Soo... 

Hope it works out.

[Glenwing]

If a method of community error reporting is being discussed, I really think a public issue tracker would be appropriate here. The problem with email or form submissions is that everyone is on a private line and can't see what anyone else has been reported, so any time there is an error there will be hundreds of duplicate reports, which will just lead to way more overhead than necessary in trying to keep on top of it all.

[moatmote]

13 minutes ago, Glenwing said:

If a method of community error reporting is being discussed, I really think a public issue tracker would be appropriate here. The problem with email or form submissions is that everyone is on a private line and can't see what anyone else has been reported, so any time there is an error there will be hundreds of duplicate reports, which will just lead to way more overhead than necessary in trying to keep on top of it all.

This so much. The "ECC Squad" is so out of touch with what is actually needed here. Not just for the reason you stated, but also because like I said before:

3 hours ago, moatmote said:

Reports should be publicly visible and it should be clear whether or not a report has been appropriately addressed, either via staff response justifying why it isn't an error, or by taking the proper steps to address the error.

Otherwise we have no way to measure how much action is actually being taken vs how much actionable feedback is being submitted.

 

I'm also pretty disappointed that the entire video focused so hard on errors with Labs data, which seemed to me like they were intentionally excluding errors in other content they produce. 

[jakemaster1390]

Definitely a step in the right direction. Errors will occur but do your best to catch them before it ever makes it to video and we will not know that it happened but glad to see that there is steps in place now to deal with errors once if it makes it to video.

[Zodiark1593]

I'd probably also rank videos differently in the QC process, according to content and goal. Videos directed primarily for entertainment with relatively low stakes (where errors won't direct potential buyers, or wrongly cast a product in an incorrect light), for example. can probably use somewhat looser criteria in certain areas while reviews, and anything containing performance data, will be subject to the most strict criteria.

 

At least where I work, this reduces workload on the QC end, while allowing ample time for thorough checks and tests of the more critical products.

[picky33]

I worry your “Very High Severity” action can turn into a slippery slope. If I watch a video from years ago I should be able to understand the information may not be the most of up to date and there could be wrong or not apply to modern day wisdom. I don’t want LTT to start deleting history because it’s no accurate. Technology is a fast paced ever changing field. 
 

to correct I suggest defining the “very high severity” better, make it less broad. 

[Drazil100]

3 hours ago, picky33 said:

I worry your “Very High Severity” action can turn into a slippery slope. If I watch a video from years ago I should be able to understand the information may not be the most of up to date and there could be wrong or not apply to modern day wisdom. I don’t want LTT to start deleting history because it’s no accurate. Technology is a fast paced ever changing field. 
 

to correct I suggest defining the “very high severity” better, make it less broad. 

I don't think that's the idea. I don't expect LMG to delete content that was at it's time completely accurate. I am sure they only intend to act on content that they straight up screwed up on.

[Error 52]

10 hours ago, FlyingPotato_is_taken said:

@LinusTech What about editing errors like the audio out of sync to the video by a fraction of a second, enough to be very annoying as pronunciation no longer matches video?

These problems have occurred dozens of times in the past with LTT and are so annoying that you can't watch the video.

 

I think that's a YouTube bug. One time I had that happen and then I downloaded the video and the lip syncing was fine. No idea why I've only really seen it with LTT though, maybe it's a side-effect of editing videos after they go up?

 

Anyway, one way to handle asterisks that I think would be ideal is to add an audio cue every time they come up. Might get a little annoying but I think it'd be for the best.

[Guest]

How about creating an "Report Errors" page on the forum here so users, if they find discrepancies with their own test benches ,can report their findings. 

[Otman]

10 hours ago, NovaNightmare said:

I know for a fact I've heard Linus say he reads all of the comments, so I'm not sure where they don't read the comments comes from. 

on the wan show, I cant find it

 

He called YouTube comments horrible and nobody read them.

[Nathat23]

As I understand the ECC is for before the video is published - how does that work with NDAs for product launches?

[Arneun]

7 hours ago, moatmote said:

 

Currently they have nothing. People in this thread keep suggesting a Google Form, which I am vehemently against for the reason I outlined in my comment:

 

Umm... no? They have relatively well organized public faced way of receiving feedback about videos, that is even relatively well moderated, and we are kinda using it right now (if anyone didn't caught that - it forum, and it like suggesting alternative while using something that works perfectly).

[Kisai]

13 hours ago, FlyingPotato_is_taken said:

@LinusTech What about editing errors like the audio out of sync to the video by a fraction of a second, enough to be very annoying as pronunciation no longer matches video?

These problems have occurred dozens of times in the past with LTT and are so annoying that you can't watch the video.

 

This tends to be the youtube "web client" and is seen when not watching the video at the intended resolution. This is because Youtube sends the video and audio as separate fragments. If you download the video with an external tool AT the intended resolution, you'll usually find it's in sync.

 

The only time I've ever seen a youtube video not be in sync is when ingress is '"too hard" for youtube's encoder, like uploading near-lossless video. This is, because, in my opinion, youtube actually encodes streams on the fly, and if nobody is watching the video, it throws it away, or pushes it to cold storage (leading to long-starts if the video wasn't prepared with the moov atom as a mov/mp4.)

 

That  isn't saying this hasn't actually happened, but it doesn't seem to as egregious as an entire video being out of sync. I can only think of maybe one or two times where I've seen this, and in my mind I think it was probably a chroma-key effect.

[Guest]

14 hours ago, LinusTech said:

Internally this is a living document, and may change as we move forward, but as part of our push for further transparency, we are publishing it in its current form to help our community better-understand how we classify errors and what action we will be taking to rectify them.

 

Types of errors

• Flubs: the host simply misspoke

• Incongruencies: information within the video is not in agreement (e.g. host says “$45” while the product page shows “$47”

• Bad spec: a table or MOGRT contains incorrect information (e.g. “48MB” instead of “96MB”

• Bad data: a graph or visualization contains data (generated by us) that is erroneous, misplaced, or spurious. 

• Bad info: a statement or representation in the video that is misleading or factually inaccurate

Types of error correction

Proactive (before the video is posted)

• Video pickup: the segment is re-shot with both audio and video

• Human audio pickup: new audio is recorded for the segment (visuals have coverage)

• AI audio pickup: new voiceover is generated. In the best case this uses a voice model that sounds like the host. However, using a robot voice  can still work.

• Text on screen: the editor overlays clarifying or correcting text

  • This will not be received by viewers who are not (or cannot) watching the screen

Reactive (after the video is posted)

• Video Replace: replacing the video with a new version without re-uploading

  • This relies on YouTube and takes some time. There are fairly strict guidelines around the use of this tool and strong justification must be provided for all changes. This is preferred to a re-upload, but there is likely a soft-limit on how often we can use this resource.

• Re-upload the video: Set the original, erroneous video to Private and upload a new version. This will have algorithmic effects, but must be done if replacement is not an option

• Pinned comment: add a comment describing the correction. 

  • This will not be received by viewers who do not check the comments (common, especially for those watching on a Smart TV) and should only be used for low-severity errors.

 

Assessing & Responding to Errors

If any of the following types of errors are discovered prior to filming (e.g. during script review) , fix them before the shoot.

 

1. Very Low Severity

- The statement could possibly be misunderstood, but it’s generally true and most people would be fine with how it’s currently presented.

- eg. The host says, “One of DisplayPort’s main advantages over HDMI is its higher bandwidth,” but this is only true when comparing certain generations of the standards. HDMI 2.1, for example, has much higher bandwidth than DP 1.1.

 

Action: no action

 

2. Low Severity

- The statement is incorrect, and we should try to clarify it better in the future, but we can leave it for now because it doesn’t meaningfully impact a viewer’s purchase decision or general knowledge.

- eg. The host says, “This monitor features moderate pixel response times and contrast because it uses an IPS panel”, but the panel manufacturer is actually AUO, rather than LG, so it’s not an IPS panel - a term which is trademarked by LG - but rather an IPS-type or IPS-like panel.

 

Action: Pinned comment

 

3. Medium Severity

- The statement is incorrect, and a minority of the audience could be upset or misled, even if it doesn’t really affect the outcome of the product evaluation for the majority.

- eg. The host says, “This GPU has AMD’s AMF video encoder, which is basically the same as NVIDIA’s NVENC,” but while the stated functionality is the same, NVENC offers better image quality, which could be important to a small minority of buyers.

 

Action: Video replaced if possible according to YouTube policy, otherwise pinned comment and above fold mention of the issue in the video description

 

4. High Severity

- The statement is incorrect, and a significant portion of the audience could be misled or make a poorly-informed purchase decision.

- eg. The host says, “This is a great gaming monitor,” when in fact, the monitor is a 60Hz VA display that, while ‘fine’ for playing games, lacks important gaming-centric features that could be easily had for the same price, making it a poor choice. 

 

Action: Video replaced if possible according to YouTube policy. Otherwise, set unlisted and disable monetization until corrective action can be taken. 

 

5. Very High Severity

- The statement is VERY incorrect—basically the opposite of correct. There is no charitable interpretation that could make this anything but completely irresponsible.

- eg. The host says, “Most games require only one high-performance core to perform their best,” when that hasn’t been true for over a decade.

 

Action: Video removed or remade

Could you elaborate with an example? Where will the Billet Lab's video be categorized?

[kobutri]

I don't think this goes anywhere far enough.

All factual errors need a correction in the video itself. Be it replacement or reupload. A pinned comment is never enough imho. It's up to you to improve your internal review processes to a point where errors become so rare that you don't trigger any of the replacement restrictions on YouTube.

A pinned comment should only ever be used to provide information that wasn't available at the time of shooting the video, e.g. price information.

The example described here as very low severity shouldn't be considered as such. HDMI 2.1, which is commonly available on modern displays and graphics cards has more bandwidth and DP 1.4, which is the newest commonly used DP standard. If a statement is general in nature, it needs to apply generally. For me this constitutes a factual error and should result in an in video correction. But even if DP had more bandwidth than HDMI, this statement shouldn't make it through review, if it doesn't explain why more bandwidth is better.

If a video has severe errors that impacts its conclusion a replacement is not enough. A separate corrections video should be issued, so people have a chance to get notified of the corrections. The corrected version both the in place one and the new one should show in their title and thumbnail that it's a correction. The video also shouldn't be changed, but simply be preceded by the corrections. 

[Vishera]

I have an issue with fixing a mistake with text on the video or a comment under the video,

This should never be an option.

Looking at how other channels deal with mistakes, corrections and updates - They just replace the segment or add to it with them delivering the corrected/updated information,

You could also do a pause in the middle of the video and deliver the correction/update through audio recording.

 

I have been part of the community since 2014 and i care about it (though i only registered in 2019) and want to see it coming out of situations like this stronger and better.

[Guest]

8 hours ago, moatmote said:

I still don't think a proper corrections video fits the tone of the WAN Show at all. They've ranted about far too many things they had little to no understanding of on the WAN Show (ahem S23 storage utilization ahem) for me to feel comfortable with them discussing corrections there.

I think it can still be a good avenue to discuss errors (especially serious ones), but it only makes sense if the error is caught after the video is uploaded IMHO. If it's an error that was caught internally before the video was uploaded, I don't think we need to hear about those as it will get boring and repetitive as time goes on. Only the ones that get missed after the fact are the only ones worth mentioning IMHO. If they were added to the WAN show as a segment then it probably should be the first thing they discuss before going into news topics. I also agree that for it to work it needs to stay on point and not veer off too far into random tangential things.

[perfidial]

14 hours ago, LinusTech said:

1. Very Low Severity

- The statement could possibly be misunderstood, but it’s generally true and most people would be fine with how it’s currently presented.

- eg. The host says, “One of DisplayPort’s main advantages over HDMI is its higher bandwidth,” but this is only true when comparing certain generations of the standards. HDMI 2.1, for example, has much higher bandwidth than DP 1.1.

 

@LinusTechPersonally I would add a kind of temporary limit statement (example above), because every information valid today is outdated somewhere in the future. To my understanding these rules should reflect how to deal with current issues. What would happen, for example if somewhere in the future HDMI comes up with a new version wich is much faster than all DP versions (hypothetically), would you really handle all videos which would be affected by such a market change, or would you just fix forward?

 

[moatmote]

3 hours ago, Arneun said:

Umm... no? They have relatively well organized public faced way of receiving feedback about videos, that is even relatively well moderated, and we are kinda using it right now (if anyone didn't caught that - it forum, and it like suggesting alternative while using something that works perfectly).

That's not true at all. Most errors reported here get 0 staff attention unless it gets tons of attention. Error reporting shouldn't be a popularity contest. 

[moatmote]

5 hours ago, Madlad351 said:

How about creating an "Report Errors" page on the forum here so users, if they find discrepancies with their own test benches ,can report their findings. 

As I suggested earlier:

12 hours ago, moatmote said:

I've been saying this for literally years, but it makes no sense to me that there isn't a dedicated section of the forum that is regularly monitored dedicated to reporting errors and misinformation. HELL NO to a Google Form that offers 0 transparency. Reports should be publicly visible and it should be clear whether or not a report has been appropriately addressed, either via staff response justifying why it isn't an error (after which the thread should remain open for any counterarguments), or by taking the proper steps to address the error. After either course of action, the thread can be moved to a "disputed errors" or "resolved errors" forum to keep the main reporting forum clear.

 

10 hours ago, moatmote said:

there must be transparency in error reports such that

• what has been reported
• which of those reports have been addressed
• which of those reports have been disputed by staff
• and which of those have yet to be addressed

is clearly visible to the public

 

However, @Unhelpful made a good point:

10 hours ago, Uttamattamakin said:

The forum might not be best since a lot of people will be afraid to complain.  This is not just an issue for this forum or this fandom.  A lot of people are not confident enough to walk up to the Kool Kid in Class and point out what they did wrong on their homework. 

 

For them an anonymous reporting form would be best.  They could square this circle by allowing for complaints or corrections to be anonymous while the content of the complaint or correction is public.  Something kin to what we see on Twitter Community Notes. 

 

Which is why they should just do this or something similar:

10 hours ago, moatmote said:

Hell, open a github. An issue tracker is exactly what's needed here. 

 

[Stephen Harris]

19 hours ago, LinusTech said:

Text on screen: the editor overlays clarifying or correcting text

• This will not be received by viewers who are not (or cannot) watching the screen

One way I've seen of addressing this, if it's major enough, is to "freeze" the video at that point and have an editor voice over explain the mistake and the correction.  Matt Parker (@standupmaths on YT) does this well with "Future Matt" correcting "Past Matt".  He's even taken the joke further where he _knows_ he'll have to add something in edit and make a specific call out to Future Matt in the original recording.

 

It does make the video a little less "flowing" so can change the look and feel, but it is an effective solution, and potentially less annoying than having a robot voice.

[Jarhead91]

I signed up just to post this here:

 

Very good and also transparent list of things you guys, and i believe this, can improve. 

 

Greets from Germany. 

[Kisai]

1 hour ago, Stephen Harris said:

One way I've seen of addressing this, if it's major enough, is to "freeze" the video at that point and have an editor voice over explain the mistake and the correction.  Matt Parker (@standupmaths on YT) does this well with "Future Matt" correcting "Past Matt".  He's even taken the joke further where he _knows_ he'll have to add something in edit and make a specific call out to Future Matt in the original recording.

 

It does make the video a little less "flowing" so can change the look and feel, but it is an effective solution, and potentially less annoying than having a robot voice.

This is only viable when it's part of the bit, and it makes video look amateurish in cases where information isn't present, or "TBA" at the time the video is produced. Nothing good happens from trying to be "first" to break a story, or review/unbox a product, because you'll have the sloppiest take. Particularly since LMG has the capacity to test hardware and software, it's not like an unboxing by j-random-internet-influencer who really has no capacity to offer to compare a product to something else they don't have.

 

So if you test X product and get results Y, but then three other hardware sites get vastly different results in their benchmarks, what does that tell you? Did you test X wrong? Is your testbench wrong? Are three other sites wrong? You're not fixing that with a voice over.

 

I'd accept "hey future host ___" from something that is speculative at the time the video is recorded due to it taking multiple days to record. That directly calls attention to the fact the host is wearing a different shirt and their hair is longer, or even filmed in a different location. But if it's just going to be an excuse for sloppy edits, then no, just get the host a wardrobe of several identical shirts, some that can be left at the office.

 

 

[camrin]

Albert Einstein was teaching a class one day, he wrote 10 simple equations on the bored. But he got one of them wrong, this was on purpose. The class was very quick to correct him and even laughed a bit at his expense. Einstein waited for everyone to become silent after witch he said, even though I got 9 out of 10 of these correct no one said good job or congratulated him. Instead they laughed and made fun of him for getting one thing wrong. Why are we all focusing on the wrong? It really seems that we all have forgotten what Linus has done for the tech community. I know I'm not a perfect person and because of that, I would never expect that from anyone. Bringing this to a close I do recognize mistake have been made, anyone who ever ran a business the size of ltt would understand they are almost impossible to stop completely. What's sets ltt apart is they actually care enough to recognize the mistake own it and move on. @LinusTech Thank you for all you have done over the years for this community!

[nlhans]

On 8/26/2023 at 12:11 AM, LinusTech said:

1. Very Low Severity

- The statement could possibly be misunderstood, but it’s generally true and most people would be fine with how it’s currently presented.

- eg. The host says, “One of DisplayPort’s main advantages over HDMI is its higher bandwidth,” but this is only true when comparing certain generations of the standards. HDMI 2.1, for example, has much higher bandwidth than DP 1.1.

 

Action: no action

 

2. Low Severity

- The statement is incorrect, and we should try to clarify it better in the future, but we can leave it for now because it doesn’t meaningfully impact a viewer’s purchase decision or general knowledge.

- eg. The host says, “This monitor features moderate pixel response times and contrast because it uses an IPS panel”, but the panel manufacturer is actually AUO, rather than LG, so it’s not an IPS panel - a term which is trademarked by LG - but rather an IPS-type or IPS-like panel.

 

Action: Pinned comment

 

3. Medium Severity

- The statement is incorrect, and a minority of the audience could be upset or misled, even if it doesn’t really affect the outcome of the product evaluation for the majority.

- eg. The host says, “This GPU has AMD’s AMF video encoder, which is basically the same as NVIDIA’s NVENC,” but while the stated functionality is the same, NVENC offers better image quality, which could be important to a small minority of buyers.

 

Action: Video replaced if possible according to YouTube policy, otherwise pinned comment and above fold mention of the issue in the video description

 

 

[..]

 

I think some of examples can have some improvements, because e.g. I don't understand why the examples of very low and low degrees are not swapped as they are now.

 

The reason I say that is because of the potential scope and the half-life time of a statement. I think these 2 aspects are missing in the examples and/or the criteria to selecting the severity of an error or simplification.

 

Now I know that both sounds impossible to get right - because we can't predict the future and so why limit the expression space to statements that are correct today and in the future? It would severely constrain what to say..

 

But if we take a look at e.g. "DP offers more BW than HDMI" - yes that does depend on the standard. But those standards also change. So a solution is either to add the standard version to give a more factual statement, or to learn people how to look up these properties in future products.

 

I think these kind of gross simplifications should not be underestimated of how they are potentially more damaging than misreferring a brandname for e.g. IPS display. Similarly, as an engineer, I don't really care much about LG being the sole owner of the IPS brand+patents, and whether other companies (e.g. AUO according to wikipedia) are sublicensing that technology to produce their own panels. That is really minor stuff IMO.

I say that because people may watch a HolyShit from e.g. 6 years ago on older tech, and keep referring to it as "DP is faster than HDMI" be fully convinced that is  factually correct. Who is going to go back and look up who made a panel in a monitor and whether that rightfully is called IPS? I think this type of information (understanding) sticks much more with people.

 

I think the 2 aspects I propose could also be captured better in the example of a medium severity case. NVENC etc. is just a brand name that hides several implementations of a video encoder that differ between card generations. That could change in the future.

Plus, that example also has a pitfall; namely "better image quality" can only be measured at a certain bitrate. If AMD's encoder supports a wider range of bitrates, that could still give it niche use cases. So that would make this example and its explanation potentially age badly and not be scoped correctly as well.

 

"Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime"

I don't want to memorize and learn facts, I can look up specs or test results, and don't see any point in bragging about memorizing them correctly. I'd rather want to gain more understanding of how technology works.

 

To be honest, I do foresee this fairly hard to implement fairly and correctly without having to scrutinize every script like its a masters dissertation. But I hope sharing these thoughts are helpful in the discussion.