Why are LTT videos so inconsistent with the captioning?

[aeiro]

Hi All,

 

As a deaf person I _really_ appreciate the availability of captions on most of LTT's videos over the last couple months. I'm not sure what triggered this change from earlier videos, but I am grateful for it. The strange thing is, though, that while most videos do have captions, the time it takes to upload them varies; sometimes it's instant, sometimes it takes a few hours, sometimes it takes a few days, and sometimes it just doesn't happen at all. I'm typically happy to financially support creators and content that is fully accessible to me and other Deaf/HoH people, and it's really bugging me that LTT is so close, but not quite there yet. I'm curious if there's any way to gain insight into the captioning process and whether there's a better way to give feedback (I do have some minor feedback on inaccuracies in the subtitles but they've been...better than many other channels). 

 

Side note: I'd also really love if the WAN show started doing transcripts as well. Some podcasts do have transcripts, but they seem like they're few and far between. I really enjoy podcasts (and especially video ones) but it's difficult to consume them without some accessibility features (YouTube's auto craptions are...pretty bad if you have no hearing). 

[piratemonkey]

I think I remember @JonoT(handles stuff like this iirc) saying that they're done by a 3rd party company or team. As for transcripts, given wan is not something that is hugely popular (at least not like their main videos) I doubt transcripts will happen. Or at least they wouldn't be non automated. It doesn't make much sense

[pythonmegapixel]

I'm not really sure about the answer to OP's questions.

 

What I will say is that someone REALLY needs to proof-read the captions. There are some cases where what has been typed is entirely inconsistent with what is being said, and doesn't even make any sense. This is mostly a problem with tech terminology (I've seen "PCIe" mistranscribed a few times, for instance.)

 

Given how much time is put into video production at LMG otherwise, it comes across as  a bit lazy to me that adding and proofreading the captions isn't just part of the regular video production workflow. Most of the videos are scripted anyway - it's just a matter of uploading the script to YouTube, tweaking the timings, and correcting any mistakes. Not complicated for an organisation as huge as LMG

 

I don't have hearing difficulties myself, but I do still appreciate the captions for times when I'd prefer not to have the sound on.

[aeiro]

3 minutes ago, pythonmegapixel said:

What I will say is that someone REALLY needs to proof-read the captions. There are some cases where what has been typed is entirely inconsistent with what is being said, and doesn't even make any sense. This is mostly a problem with tech terminology (I've seen "PCIe" mistranscribed a few times, for instance.)

Agreed - even things that should be obvious from context like 1080p often get mistranscribed to ten ADP (took me a few tries to understand myself). I'm sure that the actual captioning process is farmed out, but I suspect they're not getting any quality guarantees from the people they've contracted with, or if they are, nobody's enforcing them. Best for me guess is that the captions are 90-95% accurate, which is not terrible (and better than autocaptions for sure), but that still means a mistranscribed word every sentence or so. Industry standard is 99% accuracy, and I've seen 99.99% guarantees before (basically one mistranscription in the entire script). I do wish that they did the captioning as a prerequisite before uploading, it's clear that they strive for high quality video and audio, it's just frustrating that the same standards aren't applied to accessibility features as well. 

[BlueChinchillaEatingDorito]

I don't think their videos are word for word read from the teleprompter. So it's not as simple as copying lines from a document.

[Helpful Tech Witch]

11 minutes ago, BlueChinchillaEatingDorito said:

I don't think their videos are word for word read from the teleprompter. So it's not as simple as copying lines from a document.

Sometimes youtube channels have the CC from the teleprompter, but the video is changes slightly, and it makes no sense.

[BlueChinchillaEatingDorito]

1 hour ago, HelpfulTechWizard said:

Sometimes youtube channels have the CC from the teleprompter, but the video is changes slightly, and it makes no sense.

Exactly. Especially for a lot of LTT content where it's basically not done sitting in front of a camera and you have multiple hosts, it's going to be veer away from the script a lot. And for Techlinked when it's basically just memeing about news (with the butting in from Riley or James in the background), a lot of what's said isn't going to be in the script. 

[Slayerking92]

I didn't think anyone is actively adding CC,  they are just added as a "best guess" from youtube.

 

[pythonmegapixel]

19 minutes ago, Slayerking92 said:

I didn't think anyone is actively adding CC,  they are just added as a "best guess" from youtube.

 

If it's that kind, then they're always flagged as auto-generated.

 

Someone is definitely manually inputting them - just doing a crap job of it. Which frankly isn't acceptable for a channel as big as LTT.

 

If you're big enough to afford to spend thousands of $ to get a machine purely for review purposes, then you can afford to pay someone who actually knows about the subject matter to spend an hour or two ensuring that the captions are correct. No exceptions. This isn't some obscure "neat little feature" - it's an essential accessibility device.

[aeiro]

It looks like the most recent video has been captioned while the video before it has not yet been captioned on the main LinusTechTips channel. Any updates from the LTT team about this?

[poochyena]

Considering how much money they make and how short there videos are, it really shocks me to hear they don't have a good record with captions.

[Mark Kaine]

I'm pretty sure this isn't LTT, it's youtube... 

 

 

They recently banned captions made by humans and now only use their robot algorithm. 

 

Completely insane. I'd choose either another form of entertainment, or take it up with youtube, but you'll probably just get to talk with another robot. 

 

 

Greetings from 1982~

[poochyena]

3 minutes ago, Mark Kaine said:

They recently banned captions made by humans and now only use their robot algorithm. 

completely false

[Mark Kaine]

"completely" 

 

 

 

https://www.google.com/amp/s/www.theverge.com/platform/amp/2020/7/31/21349401/youtube-community-captions-deaf-creators-accessibility-google

[poochyena]

2 minutes ago, Mark Kaine said:

"completely" 

 

 

 

https://www.google.com/amp/s/www.theverge.com/platform/amp/2020/7/31/21349401/youtube-community-captions-deaf-creators-accessibility-google

yes, completely false. You can open up youtube right now and add subtitles. I just tried it myself

[pythonmegapixel]

3 hours ago, Mark Kaine said:

"completely" 

 

 

 

https://www.google.com/amp/s/www.theverge.com/platform/amp/2020/7/31/21349401/youtube-community-captions-deaf-creators-accessibility-google

Yes, you have entirely the wrong end of the stick there.

 

YouTube have turned off captions made by members of the community other than the video creator. It is still entirely possible for the creator to input captions themselves.

 

And I don't think it's acceptable that they aren't doing this in a timely manner

[aeiro]

I've noticed a couple of improvements on recent videos, but still the length of time is extremely variable, which is frustrating. I'd also really like to push for WAN show to have captioning - at least the recordings after the fact (though it'd certainly be awesome if the live shows had captioning too).

[dilpickle]

@LinusTech

 

Tag Linus and I'm sure he'll look into it.

 

This affects more than deaf people too. I like to watch with captions and I also notice how sometimes they aren't there on new videos. Plus all the mistakes.

[aeiro]

I also wanted to point out that TechLinked episodes rarely get captions quickly. However, due to the nature of TechLinked, by the time they get captions several days or so after release, they're out of date and not relevant anymore. 

[Kisai]

55 minutes ago, aeiro said:

I also wanted to point out that TechLinked episodes rarely get captions quickly. However, due to the nature of TechLinked, by the time they get captions several days or so after release, they're out of date and not relevant anymore. 

 

Unfortunately there's only two "not entirely unreasonable" ways around that, and they require changes on LMG's video pipeline to do

1) ASR - Automated Speech recognition. This has to be done straight off the microphone audio before any filtering is done. The best ASR however states a word-error rate around 5% which is about the same for humans.

2) ASR + Manual transcription for live broadcasts where the ASR puts captions into the video, and someone goes back and edits it later since ASR stumbles on voices that speak over each other.

 

Youtube's auto captions are basically ASR, but not trained on LMG's voices. So it will likely have that 5-7% WER on english, and worse in auto-translation (Speech Translation). If LMG wanted they could probably built box with a pair of Titan RTX's or similar to run ASR in-house so they could train it on their own voices, but that still requires running everything they produced through it to learn, and to get real-time captions is a bit of a crap-shoot still. 

 

Anyhow this gets back to why paying someone to manually transcribe live video might be cheaper, less effort and more accurate if the person transcribing the video actually knows the vernacular. 

[aeiro]

2 minutes ago, Kisai said:

1) ASR - Automated Speech recognition. This has to be done straight off the microphone audio before any filtering is done. The best ASR however states a word-error rate around 5% which is about the same for humans.

This is false. Humans can have WERs of 1% or even lower if they're sufficiently trained or experienced. A 5-7% WER for humans is typically only for untrained/inexperiencced people. In fact, I frequently make use of realtime stenocaptioners (AKA CART providers) in my job, and they are often able to achieve 98-99% accuracy in realtime, with highly technical/scientific terminology. I don't necessarily expect LTT to hire the best of the best, but it's clear that they invest high levels of money in their video, audio, and editing pipeline, so why not invest a (relatively) marginal amount of extra money into their captioning/accessibility pipeline?

2 minutes ago, Kisai said:

2) ASR + Manual transcription for live broadcasts where the ASR puts captions into the video, and someone goes back and edits it later since ASR stumbles on voices that speak over each other.

I strongly suspect (based on the consistent mistranscriptions that I see in their videos) that this is already the strategy they use. It's not a terrible strategy (especially for time constrained editing) but it's clear that whoever they've contracted to do it is not taking the time and care that LTT puts into the rest of their videos. 

2 minutes ago, Kisai said:

 

Youtube's auto captions are basically ASR, but not trained on LMG's voices. So it will likely have that 5-7% WER on english, and worse in auto-translation (Speech Translation). If LMG wanted they could probably built box with a pair of Titan RTX's or similar to run ASR in-house so they could train it on their own voices, but that still requires running everything they produced through it to learn, and to get real-time captions is a bit of a crap-shoot still. 

 

Anyhow this gets back to why paying someone to manually transcribe live video might be cheaper, less effort and more accurate if the person transcribing the video actually knows the vernacular. 

I don't want LTT to stop paying someone to do manual transcriptions; I still think that manual transcriptions are the best solution for almost every situation today. I just wish LTT would strive for the same level of quality for the transcriptions as they strive for the video and audio. I can just about guarantee that a skilled transcriptionist using the right pipeline would be capable of accurately transcribing the videos in less time than it takes to produce the final cut.

[Kisai]

34 minutes ago, aeiro said:

This is false. Humans can have WERs of 1% or even lower if they're sufficiently trained or experienced. A 5-7% WER for humans is typically only for untrained/inexperiencced people.

That needs to be highlighted, because the AI research suggests otherwise.

https://arxiv.org/ftp/arxiv/papers/1904/1904.12403.pdf

Quote

In this study, we used human transcribers and ASR systems to transcribe videoconferencing medical conversations. We found that the two manual transcriptions demonstrated similar quality with WER of 17.4%. This is higher than the WER of previous studies based on the standard telephone audio recording dataset where the manually transcribed WER was between 5.1% and 5.9%

 

 

34 minutes ago, aeiro said:

 

In fact, I frequently make use of realtime stenocaptioners (AKA CART providers) in my job, and they are often able to achieve 98-99% accuracy in realtime, with highly technical/scientific terminology. I don't necessarily expect LTT to hire the best of the best, but it's clear that they invest high levels of money in their video, audio, and editing pipeline, so why not invest a (relatively) marginal amount of extra money into their captioning/accessibility pipeline?

 

That is a good question, and perhaps it's just a question of getting people who know the vernacular and terminology. 

 

34 minutes ago, aeiro said:

I strongly suspect (based on the consistent mistranscriptions that I see in their videos) that this is already the strategy they use. It's not a terrible strategy (especially for time constrained editing) but it's clear that whoever they've contracted to do it is not taking the time and care that LTT puts into the rest of their videos. 

Well if Youtube's auto translations are about on par with untrained human translation (See linked article above) then that's probably it.

 

Personally I always run videos with subtitles on, regardless of the language.

[aeiro]

11 hours ago, Kisai said:

That needs to be highlighted, because the AI research suggests otherwise.

https://arxiv.org/ftp/arxiv/papers/1904/1904.12403.pdf

I'm not sure that study proves what you think it does. From the conclusion: 

Quote

We found that manual transcription significantly outperformed the automatic services...We posit that these findings could be generalized to other contexts.

I can't find anything showing that the WER of human transcriptionists are 5% or greater. It doesn't seem that the study goes into detail about it. In fact, they say 

 

Quote

We selected Manual CB as the reference transcript and completed a pairwise analysis for the remaining transcription services comparing the quality of all of the transcription services.

Which makes me think that they're confident in a very very low WER for humans. 

 

Besides, WER isn't the end-all-be-all of transcription quality. I haven't found an academic study that takes into account identification of different speakers, minor changes that have a big impact on the meaning ("I think this isn't stupid" vs "I think this is stupid" has two completely different meanings but has the same weight on WER as "I am going to be sad" vs "I am gonna be sad" which are functionally equivalent), or natural breaks/pauses in the subtitles. All of which humans do way better at than computers. 

 

LTT shouldn't be using AI for the subtitles for the same reason they don't use AI to edit videos or audio. 

 

[aeiro]

Two of the recent videos still don't have captions, despite it being several days. Why does this keep happening? It's making it very frustrating to be a LTT follower when it's a grab bag whether the videos are inclusive and accessible or not. 

 

 

[aeiro]

And the most recent video, while it does have captions added a while after uploading, about halfway through the captions are completely different from what is being said on screen. Given that I've been complaining about captioning quality issues since November and nobody seems to be listening, is there an alternative way of reporting video quality issues that is more effective?

 

I also want to point out that the verified gamer program was sold out prior to captions being added to the video. So it's kind of unfair for deaf folks like me to not even have the chance to participate.