Can you make a program that analyzes the video itself?

[GamerGry123]

There is a website Opus Clip, if you send a video there, this website analyzes the video and selects the most interesting moments.

You have to pay after the free trial period.

 

Is it possible to do the same, for example, on some programming code and make this program do the same. And it would just work on a computer.

 

[FI Fheonix]

3 minutes ago, GamerGry123 said:

There is a website Opus Clip, if you send a video there, this website analyzes the video and selects the most interesting moments.

You have to pay after the free trial period.

 

Is it possible to do the same, for example, on some programming code and make this program do the same. And it would just work on a computer.

 

Not Possible to use code profiling tools to identify the most interesting parts of code as it is subjective and depends on the person analyzing the code. If it's just you, then yes, theoretically you build something like that for yourself. But if you make it open source you are going to need to provide a disclaimer of possible Bias. It is a good idea though, you just need time for it.

[igormp]

10 minutes ago, GamerGry123 said:

There is a website Opus Clip, if you send a video there, this website analyzes the video and selects the most interesting moments.

You have to pay after the free trial period.

 

Is it possible to do the same, for example, on some programming code and make this program do the same. And it would just work on a computer.

 

Yes, you can have a machine learning model to do so, but it would computationally expensive to process it, and even worse to train it. This approach seems to be what that Opus Clip company did too.

[GamerGry123]

3 hours ago, igormp said:

Tak, możesz mieć do tego model uczenia maszynowego, ale jego przetwarzanie byłoby kosztowne obliczeniowo, a co gorsza, trenowanie go. Wydaje się, że takie podejście zastosowała również firma Opus Clip.

How powerful a computer do you need for this?

[igormp]

3 minutes ago, GamerGry123 said:

How powerful a computer do you need for this?

eh, idk, depends on how you preprocess your video. If you transcode it to a smaller resolution first then maybe you could go by with an 8gb nvidia GPU and 32gb of ram or so.

[GamerGry123]

18 minutes ago, igormp said:

eh, idk, zależy od tego, jak wstępnie przetworzysz wideo. Jeśli najpierw transkodujesz go do mniejszej rozdzielczości, być może mógłbyś przejść z procesorem graficznym Nvidia 8 GB i mniej więcej 32 GB pamięci RAM.

You say that you need such a computer for video editing? Yes, if you are editing video in the highest resolution, for example 4k and with a high bitrate. But it can be 720p quality, for example.

 

How powerful a computer you need to have for this technology that recognizes these moments from the video and selects them, for example, I upload a video and this program gives me from this moment to this moment the most interesting things in this video and then I edit it myself.

AI doesn't have to assemble the video for me, I just wanted it to show, for example, the best moments from a 1-hour movie.

 

How would you even do that? I know that ready-made libraries and frameworks are used in creating websites and if you know the basics, you can make a good website thanks to this, maybe there is something like this here too?

[Eigenvektor]

7 hours ago, GamerGry123 said:

You say that you need such a computer for video editing? Yes, if you are editing video in the highest resolution, for example 4k and with a high bitrate. But it can be 720p quality, for example.

No, he's saying you need such a computer to analyze the video. But only if you reduce it to 720p first. Otherwise, you'll need a more powerful machine to analyze it.

 

As was said above "most interesting" is subjective. You'd need to train a neural network by watching the movies yourself first, marking the interesting part, then using that to train the machine learning algorithm. Once you've trained it enough, it should be able to find parts in other videos that you might find interesting.

 

In a sense this is what the YouTube algorithm is trying to do. Based on the videos you watch, it's trying to suggest other videos that it thinks might be interesting to you. You be the judge if that has worked for you so far.

[GamerGry123]

7 hours ago, Eigenvektor said:

Nie, mówi, że potrzebny jest taki komputer do analizy wideo. Ale tylko wtedy, gdy najpierw zmniejszysz go do 720p. W przeciwnym razie będziesz potrzebować mocniejszej maszyny do analizy.

 

Jak powiedziano powyżej, „najciekawsze” jest subiektywne. Musiałbyś wytrenować sieć neuronową, najpierw samemu oglądając filmy, zaznaczając interesującą część, a następnie używając tego do trenowania algorytmu uczenia maszynowego. Po wystarczającym wyszkoleniu powinno być w stanie znaleźć części w innych filmach, które mogą Cię zainteresować.

 

W pewnym sensie właśnie to próbuje zrobić algorytm YouTube. Na podstawie filmów, które oglądasz, próbuje zasugerować inne filmy, które mogą Cię zainteresować. Ty będziesz sędzią, jeśli do tej pory to zadziałało.

Ok, what is this program and how?

[Eigenvektor]

56 minutes ago, GamerGry123 said:

Ok, what is this program and how?

There is no existing program. Other than possibly proprietary tech at Google they haven't made public.

 

You would essentially need to create/train your own neural network to recognize things you find interesting.

 

To ensure you can train the network on the proposed hardware, you would first need to reduce the videos in size. Otherwise you will need a much more powerful machine.

[GamerGry123]

19 minutes ago, Eigenvektor said:

Nie ma istniejącego programu. Poza prawdopodobnie zastrzeżoną technologią w Google, której nie upublicznili.

 

Zasadniczo musiałbyś stworzyć/wyszkolić własną sieć neuronową, aby rozpoznawać rzeczy, które uważasz za interesujące.

 

Aby upewnić się, że możesz trenować sieć na proponowanym sprzęcie, musisz najpierw zmniejszyć rozmiar filmów. W przeciwnym razie będziesz potrzebować znacznie potężniejszej maszyny.

Not something ready for open source? I mean, if you have to do it yourself, how do you do it? What code to use for this? It's done just like in programming, are you writing code? Could you do something like that?

[Avocado Diaboli]

37 minutes ago, GamerGry123 said:

Not something ready for open source? I mean, if you have to do it yourself, how do you do it? What code to use for this? It's done just like in programming, are you writing code? Could you do something like that?

How do you think the people who offer this service on the website you mentioned did it? Yeah if it truly is what you claim it is, they programmed it themselves from scratch, gathered a bunch of training data (which is a lot of work, because you need to have it tagged properly and that is still a manual process, otherwise, you wouldn't need to train a model if you can already automate it), trained their model (which is incredibly time consuming and/or expensive) that's able to determine which clips fed into it will fit the criteria for which it was trained for. So yes, you can do it, but you, personally, probably won't. 

[igormp]

54 minutes ago, GamerGry123 said:

how do you do it? What code to use for this? It's done just like in programming, are you writing code? Could you do something like that?

1- Gather your training data

   1.1- you could just scrape youtube videos and scrape their "interest" view to use as training data

2- Pre-process you data so it can be easily used my your training pipeline (stuff like reducing the video resolution and splitting it into separate images)

3- Build a model that's time/context-aware (something like a LSTM or even a transformer)

4- Train it using your previous data

5- Build a service around your model that's able to get a video as input, pre-process it, run the inference on the previous model, and attach the resulting outputs back into your final result.

 

No, there's no available open source code to do all of the above in one go, you would need to build each piece and glue them together.

[Raytsou]

Yeah considering what's "interesting" is very subjective, youd have to train your own algorithm. Yes it is just programming, but it's machine learning engineering.

 

What's you have to do is scrap together a database of a few thousand videos and manually label the "interesting" parts of each video. Then you feed train the ml algo on this data ( basically you feed the data into a bunch of linear algebra and have it shit out a function that is able to take a video and output the label). Once you have the model you're now ready to run it on newer videos and it should hopefully output what you want. You don't need a GPU at all if you're not doing this real time, you can set a video to be processed and come back a few days later to the labels generated.

 

[igormp]

35 minutes ago, Raytsou said:

Yeah considering what's "interesting" is very subjective, youd have to train your own algorithm. Yes it is just programming, but it's machine learning engineering.

 

What's you have to do is scrap together a database of a few thousand videos and manually label the "interesting" parts of each video. Then you feed train the ml algo on this data ( basically you feed the data into a bunch of linear algebra and have it shit out a function that is able to take a video and output the label). Once you have the model you're now ready to run it on newer videos and it should hopefully output what you want. You don't need a GPU at all if you're not doing this real time, you can set a video to be processed and come back a few days later to the labels generated.

 

Training without a GPU is suicide tho, even if you can manage to go by on inference without one.

[Eigenvektor]

4 hours ago, GamerGry123 said:

Not something ready for open source? I mean, if you have to do it yourself, how do you do it? What code to use for this? It's done just like in programming, are you writing code? Could you do something like that?

You mean for machine learning in general? Frameworks for that exist. But your use case in particular? I would be very surprised if something like that already exists on the open market.

 

You have to consider that video clips that you find interesting might be extremely boring to me and vice versa. So an existing neural network trained by a single person is most likely not going to be useful to anyone else.

 

Training such an algorithm would also take a very long time. You would have to watch thousands of hours of video, mark sections of it as interesting and not interesting and then use that data to train an AI, before the AI would be able to do what you want it to.

 

That is only really doable by crowd sourcing the whole process. Which means you need a platform like YouTube. They have hundreds of millions of viewers watching billions of videos. So they can quickly gather information which videos get watched the most and which sections of a video are watched the most and so on.

 

This also makes the results more generalized, because the AI is trained by the preferences of a lot of people. So not only do they get results like "this is interesting to 80% of our viewers" but also "this is interesting to 75% of male viewers, aged between 20–25 who also like videos in these other categories" etc.

[igormp]

9 minutes ago, Eigenvektor said:

You have to consider that video clips that you find interesting might be extremely boring to me and vice versa. So an existing neural network trained by a single person is most likely not going to be useful to anyone else.

 

Training such an algorithm would also take a very long time. You would have to watch thousands of hours of video, mark sections of it as interesting and not interesting and then use that data to train an AI, before the AI would be able to do what you want it to.

You can also just scrape this data from youtube itself

[Eigenvektor]

Just now, igormp said:

You can also just scrape this data from youtube itself

Sure, provided Google doesn't block you when you do that a lot