LTT Video Error Handling Standard Operating Procedure (SOP)

[cmndr]

4 minutes ago, Ryz said:

Do you have any new processes for catching these errors?

I'm going to take a guess and say that there ARE processes, it's just that execution isn't perfect. 

Part of continuous process improvement is making sure that those processes are well executed. 

[joshfrog]

Something else you should look at doing is publishing a record of the fixes.

To a forum post or a page somewhere so a record can be viewed with corrections.

 

If someone watched a video that got removed or changed that can view what video was changed, what the issue was, how it was changed.

If a video gets replaced or removed there should be a record of it so if anyone watched the wrong version they can go back and see they watched a video that got removed or replaced and the info they are basing there knowledge on could be wrong.

 

[robrobbierobertson]

18 minutes ago, Ryz said:

Do you have any new processes for catching these errors?

They talked before about posting the videos early for peer review.
This is a bad idea.
Instead of posting the entire video, why not just post the graphs and basic tech info (from the script) for folks to review.
It's much easier for folks to scrutinize just the data, rather than having to watch a 15 minute video and try and pull the info themselves.
And there's still a reason to watch the video and no chance of it being shared.
 

[EllieCat]

Lingua may have a sense of quantity over quality.

[robrobbierobertson]

Just now, joshfrog said:

Something else you should look at doing is publishing a record of the fixes.

To a forum post or a page somewhere so a record can be viewed with corrections.

 

If someone watched a video that got removed or changed that can view what video was changed, what the issue was, how it was changed.

If a video gets replaced or removed there should be a record of it so if anyone watched the wrong version they can go back and see they watched a video that got removed or replaced and the info they are basing there knowledge on could be wrong.

 

Or better yet, post a video once a month or once every 2 months just as a 'update, here's what we got wrong, and here is how  we got it wrong' or 'here's what's changed since we posted the video'.
People won't go to a page to find info. They just won't. BUT they will watch a video (even if it's only 5 minutes long) with new info. Plus it can be monetized so there is incentive to do so.

[Branmo]

Honestly I love this

[Nene]

1 minute ago, robrobbierobertson said:

They talked before about posting the videos early for peer review.
This is a bad idea.
Instead of posting the entire video, why not just post the graphs and basic tech info (from the script) for folks to review.
It's much easier for folks to scrutinize just the data, rather than having to watch a 15 minute video and try and pull the info themselves.
And there's still a reason to watch the video and no chance of it being shared.
 

Posting the info for review is a good idea for exactly the reason you stated, but that doesn't catch errors that are in the script instead of the graphs, so I think having both is the best way to go.

 

It's also a good idea to have a recognised and official way to report errors we find, something we know is being read and comes with some sort of acknowledgement that it's been received and a person is actually going to look at it. That's starting to sound a lot like a bug tracker isn't it.

[Squilliam]

1 hour ago, LinusTech said:

Internally this is a living document, and may change as we move forward, but as part of our push for further transparency, we are publishing it in its current form to help our community better-understand how we classify errors and what action we will be taking to rectify them.

 

Types of errors

• Flubs: the host simply misspoke

• Incongruencies: information within the video is not in agreement (e.g. host says “$45” while the product page shows “$47”

• Bad spec: a table or MOGRT contains incorrect information (e.g. “48MB” instead of “96MB”

• Bad data: a graph or visualization contains data (generated by us) that is erroneous, misplaced, or spurious. 

• Bad info: a statement or representation in the video that is misleading or factually inaccurate

Types of error correction

Proactive (before the video is posted)

• Video pickup: the segment is re-shot with both audio and video

• Human audio pickup: new audio is recorded for the segment (visuals have coverage)

• AI audio pickup: new voiceover is generated. In the best case this uses a voice model that sounds like the host. However, using a robot voice  can still work.

• Text on screen: the editor overlays clarifying or correcting text

  • This will not be received by viewers who are not (or cannot) watching the screen

Reactive (after the video is posted)

• Video Replace: replacing the video with a new version without re-uploading

  • This relies on YouTube and takes some time. There are fairly strict guidelines around the use of this tool and strong justification must be provided for all changes. This is preferred to a re-upload, but there is likely a soft-limit on how often we can use this resource.

• Re-upload the video: Set the original, erroneous video to Private and upload a new version. This will have algorithmic effects, but must be done if replacement is not an option

• Pinned comment: add a comment describing the correction. 

  • This will not be received by viewers who do not check the comments (common, especially for those watching on a Smart TV) and should only be used for low-severity errors.

 

Assessing & Responding to Errors

If any of the following types of errors are discovered prior to filming (e.g. during script review) , fix them before the shoot.

 

1. Very Low Severity

- The statement could possibly be misunderstood, but it’s generally true and most people would be fine with how it’s currently presented.

- eg. The host says, “One of DisplayPort’s main advantages over HDMI is its higher bandwidth,” but this is only true when comparing certain generations of the standards. HDMI 2.1, for example, has much higher bandwidth than DP 1.1.

 

Action: no action

 

2. Low Severity

- The statement is incorrect, and we should try to clarify it better in the future, but we can leave it for now because it doesn’t meaningfully impact a viewer’s purchase decision or general knowledge.

- eg. The host says, “This monitor features moderate pixel response times and contrast because it uses an IPS panel”, but the panel manufacturer is actually AUO, rather than LG, so it’s not an IPS panel - a term which is trademarked by LG - but rather an IPS-type or IPS-like panel.

 

Action: Pinned comment

 

3. Medium Severity

- The statement is incorrect, and a minority of the audience could be upset or misled, even if it doesn’t really affect the outcome of the product evaluation for the majority.

- eg. The host says, “This GPU has AMD’s AMF video encoder, which is basically the same as NVIDIA’s NVENC,” but while the stated functionality is the same, NVENC offers better image quality, which could be important to a small minority of buyers.

 

Action: Video replaced if possible according to YouTube policy, otherwise pinned comment and above fold mention of the issue in the video description

 

4. High Severity

- The statement is incorrect, and a significant portion of the audience could be misled or make a poorly-informed purchase decision.

- eg. The host says, “This is a great gaming monitor,” when in fact, the monitor is a 60Hz VA display that, while ‘fine’ for playing games, lacks important gaming-centric features that could be easily had for the same price, making it a poor choice. 

 

Action: Video replaced if possible according to YouTube policy. Otherwise, set unlisted and disable monetization until corrective action can be taken. 

 

5. Very High Severity

- The statement is VERY incorrect—basically the opposite of correct. There is no charitable interpretation that could make this anything but completely irresponsible.

- eg. The host says, “Most games require only one high-performance core to perform their best,” when that hasn’t been true for over a decade.

 

Action: Video removed or remade

I love the changes. As someone (with ADHD) who struggles with fine detail during repeated work, I would struggle to implement this. As a regular and long time viewer, I super appreciate it. Keep up the good work!

[joshfrog]

1 minute ago, robrobbierobertson said:

Or better yet, post a video once a month or once every 2 months just as a 'update, here's what we got wrong, and here is how  we got it wrong' or 'here's what's changed since we posted the video'.
People won't go to a page to find info. They just won't. BUT they will watch a video (even if it's only 5 minutes long) with new info. Plus it can be monetized so there is incentive to do so.

I did think about that being a video production company and its not a bad idea, my thinking was however it adds a delay to making it clear what the error was and that there was an error. If they made a video on the 1st of a month and make a video at the end of a month that's 30 days before people know a mistake was made.

A post with 3 lines of text that could be pulled from an email explaining to the video editor what the issue was and what the fix needs to be,

Just copy and paste into a document ASAP that can be viewed anytime, takes 1min to fill out once the error is know.  

[Squilliam]

1 hour ago, Absentia13 said:

The AI voice thing is ridiculous. Honestly, it's literally a 10 second section or something just get the host to record it again or delay the video. 

I disagree. There may be instances where a host is unavailable (vacation, etc), but the video needs to be uploaded on a specific date (sponsor requirements or NDA ending). This makes it so the host can enjoy their time off, and the business isn’t placed at a standstill. Using AI keeps the same host in a sense, so it doesn’t feel as jarring of a transition.

[robrobbierobertson]

2 minutes ago, Nene said:

Posting the info for review is a good idea for exactly the reason you stated, but that doesn't catch errors that are in the script instead of the graphs, so I think having both is the best way to go.

 

It's also a good idea to have a recognised and official way to report errors we find, something we know is being read and comes with some sort of acknowledgement that it's been received and a person is actually going to look at it. That's starting to sound a lot like a bug tracker isn't it.

Honestly I think relying on users is a terrible idea. But if they're going to do it, my way is probably the best.
I speak from a position of authority in this area. I have a semi-tech related site that gets around 200k folks a day (attached is my google analytics from today). I decided to crowd source some of the information. The first week there was a lot of interest (which means around 2k views of the page and about 100 comments) and now it goes weeks with only 1 or 2 new comments.
Not to mention the users have no real investment in accuracy and will not be as good as a staff review. I know. I've seen it.

[Joshatron]

I feel like it would be a good addition for High and Very High Severity that a correction is made on the WAN show for the week that the error is noticed. That would be a great way to inform the community that had already watched the episode. Maybe a community post for those also?

[NovaNightmare]

1 hour ago, Otman said:

What if users want to report issues/errors you guys made?

 

Based on my understanding,you guys don't read YouTube comments, so if you guys miss an error, How do we report it

I know for a fact I've heard Linus say he reads all of the comments, so I'm not sure where they don't read the comments comes from. 

[Absentia13]

6 minutes ago, Squilliam said:

I disagree. There may be instances where a host is unavailable (vacation, etc), but the video needs to be uploaded on a specific date (sponsor requirements or NDA ending). This makes it so the host can enjoy their time off, and the business isn’t placed at a standstill. Using AI keeps the same host in a sense, so it doesn’t feel as jarring of a transition.

In my opinion, and it's just my opinion....I don't ever want AI anything in channels I watch. It's silly and stupid and inauthentic. 

[robrobbierobertson]

6 minutes ago, joshfrog said:

I did think about that being a video production company and its not a bad idea, my thinking was however it adds a delay to making it clear what the error was and that there was an error. If they made a video on the 1st of a month and make a video at the end of a month that's 30 days before people know a mistake was made.

A post with 3 lines of text that could be pulled from an email explaining to the video editor what the issue was and what the fix needs to be,

Just copy and paste into a document ASAP that can be viewed anytime, takes 1min to fill out once the error is know.  

Or do both. Have the document for immediate dissemination, but also do a video so people will actually view it. I mean, if they add a top pinned comment even after 6 hours... that's potentially millions of people who saw the wrong info and will most likely never re-view the video. How often do you re-view a video you saw earlier in the day? 
 

[Ethnod]

15 minutes ago, joshfrog said:

Something else you should look at doing is publishing a record of the fixes.

To a forum post or a page somewhere so a record can be viewed with corrections.

I really like this idea, great suggestion

[Azzychan]

17 minutes ago, robrobbierobertson said:

Or better yet, post a video once a month or once every 2 months just as a 'update, here's what we got wrong, and here is how  we got it wrong' or 'here's what's changed since we posted the video'.
People won't go to a page to find info. They just won't. BUT they will watch a video (even if it's only 5 minutes long) with new info. Plus it can be monetized so there is incentive to do so.

The channel, Extra Credits, does this when they get stuff wrong during a period of History and such.

[MartinTheActor]

Replacement of videos is a line for me. Despite the ability for someone to do them - they simply shouldn't. At bare minimum a correction should be put out visibly and the video removed. When someone can replace a video, and if I'm watching on Android TV, Fire, Roku, Smart TV etc. I have no idea if I am watching an original or replacement video. So unless part of the process would be to have in the first few seconds a VO stating that this video was replaced due to x, y, or z this does very little to repair any of the trust.

In all seriousness there are smaller Youtubers who do error correction better. They carry it out in a way that trounces this process. Look outside of the tech sphere and look at how other info-tainment manages to error correct. Don't do the thing of pretending that tech is it's only precious thing - it isn't. It's a theme, and you're a media production company if you like it or not. Look to other media production companies to see how they handle these processes.

 

See the following example:

Corrections and Clarifications « Tom Scott

 

Edited August 26, 2023 by MartinTheActor
Edited to add an example

[robrobbierobertson]

2 minutes ago, MartinTheActor said:

Replacement of videos is a line for me. Despite the ability for someone to do them - they simply shouldn't. At bare minimum a correction should be put out visibly and the video removed. When someone can replace a video, and if I'm watching on Android TV, Fire, Roku, Smart TV etc. I have no idea if I am watching an original or replacement video. So unless part of the process would be to have in the first few seconds a VO stating that this video was replaced due to x, y, or z this does very little to repair any of the trust.

In all seriousness there are smaller Youtubers who do error correction better. They carry it out in a way that trounces this process. Look outside of the tech sphere and look at how other info-tainment manages to error correct. Don't do the thing of pretending that tech is it's only precious thing - it isn't. It's a theme, and you're a media production company if you like it or not. Look to other media production companies to see how they handle these processes.

 

See the following example:

Corrections and Clarifications « Tom Scott

 

3.9 million views. Seems like it's a good idea to issue corrections. 

Hmm... but this could lead to adding intentional errors JUST so you have corrections to make! And then if you do errors in your correction, you have to do ANOTHER correction video! 
By the year 2025, 82% of all videos on youtube will be correction videos. 

[MartinTheActor]

Just now, robrobbierobertson said:

3.9 million views. Seems like it's a good idea to issue corrections. 

Hmm... but this could lead to adding intentional errors JUST so you have corrections to make! And then if you do errors in your correction, you have to do ANOTHER correction video! 
By the year 2025, 82% of all videos on youtube will be correction videos. 

My point really was that Tom Scott is one of the best for citing sources, doing the research and being honest when he's made a mistake. All this outline on the OP proves is that LMG haven't bothered to look outside of their little sphere. Going back to Tom Scott, well let's just say he gets views without clickbait thumbnails...he's also branched out and experimented frequently...I don't think he'd be doing stuff for the views.

[MadameKate]

Looks like a sensible approach. :⁠-⁠)

[Nene]

26 minutes ago, robrobbierobertson said:

Honestly I think relying on users is a terrible idea.

Relying on them? Yep, terrible idea. With enough eyeballs most problems are shallow though, and a lot of folk want to help. You also gain access to people who might be experts in fields that simply don't exist in the LTT staff.

[joshfrog]

This post should be in the Featured Topics area on the side of the page so others can see it and review it.

[ETHANO8225]

New account but ive been watching since like 2016. This really should've been in place earlier w/o this drama, regardless, it seems very solid. I'd much prefer an AI voice tell me correct information rather than Linus tell me objectively false info, 200% of the time.

 

I also think it would be very important to somehow highlight the comment-section edits. Like, at the top of the description, it should have [EDITS IN COMMENTS] or something of the sort so I (AND OTHER CONSUMERS) don't have to go digging wherever to find it. However it's done, it must be done OBVIOUSLY. If the viewer misses the edit, what's the point of fixing it in the first place? The mistake would still influence the viewer negatively, without them even realizing you fixed it. The reason I say all of this, is a LOT of people don't even bother looking at the comments, but occasionally peek near the title. An obvious warning would probably make them look at the comments for the fix. This is what I mean

 

All in all though, this seems very solid. Follow. Through.

Edited August 26, 2023 by ETHANO8225
spacing & adding a line or 2, adding example

[SansVarnic]

10 minutes ago, joshfrog said:

This post should be in the Featured Topics area on the side of the page so others can see it and review it.

Its a Pinned topic in the LTT Subforum, its already pretty much a featured topic.