LTT Video Error Handling Standard Operating Procedure (SOP)

[LinusTech]

Internally this is a living document, and may change as we move forward, but as part of our push for further transparency, we are publishing it in its current form to help our community better-understand how we classify errors and what action we will be taking to rectify them.

 

Types of errors

• Flubs: the host simply misspoke

• Incongruencies: information within the video is not in agreement (e.g. host says “$45” while the product page shows “$47”

• Bad spec: a table or MOGRT contains incorrect information (e.g. “48MB” instead of “96MB”

• Bad data: a graph or visualization contains data (generated by us) that is erroneous, misplaced, or spurious. 

• Bad info: a statement or representation in the video that is misleading or factually inaccurate

Types of error correction

Proactive (before the video is posted)

• Video pickup: the segment is re-shot with both audio and video

• Human audio pickup: new audio is recorded for the segment (visuals have coverage)

• AI audio pickup: new voiceover is generated. In the best case this uses a voice model that sounds like the host. However, using a robot voice  can still work.

• Text on screen: the editor overlays clarifying or correcting text

  • This will not be received by viewers who are not (or cannot) watching the screen

Reactive (after the video is posted)

• Video Replace: replacing the video with a new version without re-uploading

  • This relies on YouTube and takes some time. There are fairly strict guidelines around the use of this tool and strong justification must be provided for all changes. This is preferred to a re-upload, but there is likely a soft-limit on how often we can use this resource.

• Re-upload the video: Set the original, erroneous video to Private and upload a new version. This will have algorithmic effects, but must be done if replacement is not an option

• Pinned comment: add a comment describing the correction. 

  • This will not be received by viewers who do not check the comments (common, especially for those watching on a Smart TV) and should only be used for low-severity errors.

 

Assessing & Responding to Errors

If any of the following types of errors are discovered prior to filming (e.g. during script review) , fix them before the shoot.

 

1. Very Low Severity

- The statement could possibly be misunderstood, but it’s generally true and most people would be fine with how it’s currently presented.

- eg. The host says, “One of DisplayPort’s main advantages over HDMI is its higher bandwidth,” but this is only true when comparing certain generations of the standards. HDMI 2.1, for example, has much higher bandwidth than DP 1.1.

 

Action: no action

 

2. Low Severity

- The statement is incorrect, and we should try to clarify it better in the future, but we can leave it for now because it doesn’t meaningfully impact a viewer’s purchase decision or general knowledge.

- eg. The host says, “This monitor features moderate pixel response times and contrast because it uses an IPS panel”, but the panel manufacturer is actually AUO, rather than LG, so it’s not an IPS panel - a term which is trademarked by LG - but rather an IPS-type or IPS-like panel.

 

Action: Pinned comment

• Note: Pinned comment text must be approved by the Community Coordinator before being posted.

 

3. Medium Severity

- The statement is incorrect, and a minority of the audience could be upset or misled, even if it doesn’t really affect the outcome of the product evaluation for the majority.

- eg. The host says, “This GPU has AMD’s AMF video encoder, which is basically the same as NVIDIA’s NVENC,” but while the stated functionality is the same, NVENC offers better image quality, which could be important to a small minority of buyers.

 

Action: Video replaced if possible according to YouTube policy, otherwise pinned comment and above fold mention of the issue in the video description

• If doing a video replace, a pinned comment should communicate this is coming and note the expected changes. This comment should be un-pinned when the video replace is complete. 

• Note: Pinned comment text must be approved by the Community Coordinator before being posted.

 

4. High Severity

- The statement is incorrect, and a significant portion of the audience could be misled or make a poorly-informed purchase decision.

- eg. The host says, “This is a great gaming monitor,” when in fact, the monitor is a 60Hz VA display that, while ‘fine’ for playing games, lacks important gaming-centric features that could be easily had for the same price, making it a poor choice. 

 

Action: Video replaced if possible according to YouTube policy. Otherwise, set unlisted and disable monetization until corrective action can be taken. 

• If doing a video replace, a pinned comment should communicate this is coming and note the expected changes. This comment should be un-pinned when the video replace is complete. 

• Note: Pinned comment text must be approved by the Community Coordinator before being posted.

 

5. Very High Severity

- The statement is VERY incorrect—basically the opposite of correct. There is no charitable interpretation that could make this anything but completely irresponsible.

- ex. The host says, “Most games require only one high-performance core to perform their best,” when that hasn’t been true for over a decade.

 

Action: Video removed or remade

Edited September 12, 2023 by LinusTech
SOP Updated

[DiamondDaveL]

Looks very reasonable and thorough to me. I imagine it will be hard to exactly see the effects of new processes and controls as when something is fixed before it's public, we (the viewers) won't know about it. 

 

EDIT: I suggested this below in response to another comment, but how should viewers report errors? Maybe LMG can setup a google for that requires an email to prevent spam, and have a text box to paste a link/title for the video in question, type of error (drop-down list), and then error details textbox. 

[Ottoman420]

These all look very positive and look like correct steps for the issue of testing, reviewing and presenting a product. Its a shame that it wasn't standard practice already but better late then never.

 

I hope they are followed through all the way.

[emosun]

A step in the right direction.

[AuroraEsmeralda]

This seems like a good thing.

 

That said all examples use "eg." except for tier 5 errors which uses "ex." and it's really bugging me.

 

EDIT: TBC this isn't trying to say "this is an error requiring correction" it's "this annoyed me a little bit because it's slightly inconsistent."

[Hyruloise]

These all look like good measures to take. The most important part is that your internal processes are solid enough to be able to catch them before they go online. But your commitment to taking down videos that have errors in them is commendable. Show us what you got!

[Mister Sandman]

Progress is progress! 
Hoping everyone on staff is doing alright, can't wait to see the new improved content. Love you all at LTT. 

[Absentia13]

The AI voice thing is ridiculous. Honestly, it's literally a 10 second section or something just get the host to record it again or delay the video. 

[nitio]

10 minutes ago, AuroraEsmeralda said:

This seems like a good thing.

 

That said all examples use "e.g." except for tier 5 errors which uses "e.x." and it's really bugging me.

That would be a very low severity, so, no action.

[Gamercave]

There seems to be a strong emphasis on audience being upset from my reading considering that it is continuously listed first. more emphasis on factual than audience opinion of it, if your responding to the audience your always going to be behind the ball get out infront and put the focus on the fact rather than the opinion 

[FlyingPotato_is_taken]

@LinusTech What about editing errors like the audio out of sync to the video by a fraction of a second, enough to be very annoying as pronunciation no longer matches video?

These problems have occurred dozens of times in the past with LTT and are so annoying that you can't watch the video.

 

[Otman]

What if users want to report issues/errors you guys made?

 

Based on my understanding,you guys don't read YouTube comments, so if you guys miss an error, How do we report it

[Smeeoh]

4 minutes ago, FlyingPotato_is_taken said:

@LinusTech What about editing errors like the audio out of sync to the video by a fraction of a second, enough to be very annoying as pronunciation no longer matches video?

These problems have occurred dozens of times in the past with LTT and are so annoying that you can't watch the video.

 

When was the last time that happened? 

[DiamondDaveL]

3 minutes ago, Otman said:

What if users want to report issues/errors you guys made?

 

Based on my understanding,you guys don't read YouTube comments, so if you guys miss an error, How do we report it

This comment is much more on topic than the one you made earlier. 

 

@LinusTech, may I suggest an email tip line or google form to report errors? Google Forms would probably be pretty easy on both sides, reporting and managing reports. 

[VictorAlvarenga]

Perhaps one thing that may also help catching errors proactively is to release the video a few hours (maybe a day) early on Floatplane and see if the public finds any inconsistencies, thus providing a safety margin before releasing the video to a much larger audience on Youtube. (Note: I am not a Floatplane subscriber)

[Terrordodo]

Great, this seems like progress establishing a proper process for error handling. A few thing I would mention, however, about the topic of responsibilities and timelines. This might be omitted to the public here and might be included in the internal SOP but this needs to be clarified somewhat anyway. 

 

Regarding timelines:

When are the corrective actions implemented? Are there processes for when this deadline cannot be met? For example, how would a medium severity error escalate in corrective actions when not properly in time?

 

Regarding responsibilities:

Who is monitoring this process? Are responsibilities assigned for each error or is there a general responsible employee taking care of errors? Who needs to be involved for which errors or can every level of error be dealt with by every employee? Is this process traceable, i.e. is there the option to trace back who corrected which error in what way when?

 

This process should be really detailed in how you handle errors and you should consider processes for avoiding these in the first place. Considering the lab, is there a thought towards implementing a QMS? I would think this would increase the confidence in anything the lab does drastically.

[crism]

@LinusTech thank you for sharing - this all sounds very reasonable.

 

any updates as to when the new screwdriver color-way that Nick teased will be available for pre-orders?

[DDock]

@LinusTech I honestly hope this all results in bringing back early access to Floatplane videos, not because I actually care to see the videos, but I think it would be a HUGE benefit for you guys for us to see and critique the video prior to it being posted on YouTube. It would give time for the video to be Voiced Over or other fixes as mentioned without impacting the YouTube algorithm. Just my 2c.

[Lomac]

10 minutes ago, Absentia13 said:

The AI voice thing is ridiculous. Honestly, it's literally a 10 second section or something just get the host to record it again or delay the video. 

It might be beneficial for instances where A-roll has been finalized but the host is unavailable for a while after shooting the video due to vacation or being out of town to cover a convention or something.

[Arneun]

1 minute ago, Gamercave said:

There seems to be a strong emphasis on audience being upset from my reading considering that it is continuously listed first. more emphasis on factual than audience opinion of it, if your responding to the audience your always going to be behind the ball get out infront and put the focus on the fact rather than the opinion 

It seems to me that it's more of a "how much it will misinform the viewer" not "how much the viewer will be upset" but what will be misinforming for one part of the audience will result in upset viewer that is technically versed in that area.

Taking for example - I won't notice if they'll say DP is faster than HDMI because that's not what I am proficient in. But if by any case they'll cover new features in C# 10, or something like that - well, then I could be one that is angered by mistake. For any given error that will leak out to public (at this scale) there will be a degree of reactions - the thing is that audience upset is most quantifiable way of measuring video correctness they have, because they probably have (with probability approaching 1) enough viewers to create vocal minority for any given topic they'll cover.

[hug0lopolus]

Yup that all seems fair and logical

[Anezay]

8 minutes ago, FlyingPotato_is_taken said:

@LinusTech What about editing errors like the audio out of sync to the video by a fraction of a second, enough to be very annoying as pronunciation no longer matches video?

These problems have occurred dozens of times in the past with LTT and are so annoying that you can't watch the video.

 

Severity tier 1, no action.

[FlyingPotato_is_taken]

Just now, Smeeoh said:

When was the last time that happened? 

No idea. Stopped watching months ago.

I check from time to time and if the title isn't clickbait (some titles are so bad that you have no idea e.g. "I made an expensive mistake" is somehow a car review and not a vlog).

If the title is inserting I open it in the background for audio and tab in if their is something interesting.

[Woofer21]

22 minutes ago, AuroraEsmeralda said:

This seems like a good thing.

 

That said all examples use "e.g." except for tier 5 errors which uses "e.x." and it's really bugging me.

they are already failing! worthless! /s

[Smeeoh]

1 minute ago, FlyingPotato_is_taken said:

No idea. Stopped watching months ago.

I check from time to time and if the title isn't clickbait (some titles are so bad that you have no idea e.g. "I made an expensive mistake" is somehow a car review and not a vlog).

If the title is inserting I open it in the background for audio and tab in if their is something interesting.

Then how do you know it’s a problem? You stopped watching…and when you do it’s in the background…