Project: Plane Engine (floatplane-like side project)

[sample_text]

 

 

Hi!

 

In this post I'll be posting stuff about a vacation project I started a week ago. It's called plane engine (inspired by floatplane). I'm not trying to copy floatplane, I just wanted to try to build my own engine .

It will be fully written in NodeJS (es6). I'm thinking of improving the performance of some parts (mostly the transcoding service) with native C++ bindings.

 

What plane engine will do:

• Trans-coding media
• Serving media
• Handling metadata
• Handling extra info linked to the media (like comments, etc...)

What plane engine will NOT do:

• It won't be a full video platform. I'm probably not making a frontend for it (I just want to build the backend. Maybe I'll do something in Angular or React if I'm still interested in the project ).
• It won't handle any user account stuff
• It won't auto scale. You should probably implement this on the loadbalancer layer or use a central performance monitoring system.
• It won't handle payments, etc.. (This could be implemented in another microservice using something like Stripe, etc...)
• It won't be production ready. I don't have the resources to fully test this project without deployment. I'm not planning to use this in production.
• Idk, I'm just making this for fun

 

The architecture

The application uses a conveyor belt system design. The application is split in multiple microservices that each do it's own thing.

This allows for easier maintenance, smoother updates, easy bottleneck cancellation (spawning more instances for a microservice that takes longer than others).

Microservices allow for dynamic scaling, this is a big cost saver because not every service has to run 24/7. Also, when your site is on high load, it can spawn a couple extra services.

Splitting the application also makes is so you aren't bound to run the full app on one platform. For example: you can host the injection service on a cloud provider (like AWS or GCloud) and host the transcoding service on your own infra.

 

It's based around MongoDB. There are some reasons why I chose for this:

• It's nice to use with NodeJS
• It has GridFS (I'll talk about why this is nice). It's ok for temporary storage while processing. It's a bit slower than normal storage. But 
• It performs great from my experience
• Like almost every other database, it scales nicely

 

Here is the "path" the video takes when a creator uploads a video.

 

First, it's handled by the Injection service. This service creates an entry in the database. This entry contains the video metadata (description, title, etc...).

It also contains an unique ID that points to the video file in GridFS. I use this only for the conveyor system. Once the video is done processing it will be stored on a storage server (still have to do that service ).

It will also store a state with the entry, each service checks the database for entries with a specific state. This describes what to do next (eg. transcode, publish, etc...).

This is probably not the best solution. A better solution would be to make it event based.

The injection service doesn't do much more. It's focus is to quickly move the video into the processing queue.

 

The second service the video will reach is the Transcoding service. This service transcodes the video (who thought ). It checks the database for entries with the transcode state.

It will transcode the video to the right formats for serving to the right platforms in multiple resolutions. It currently uses a simple solution based on ffmpeg.

I'm still working on this service, I'm looking into native C++ bindings with some CUDA magic.

 

I'm still working on the rest of the services, these will probably handle the distribution of the transcoded media to multiple servers around the world.

It should update the database once it has stored the transcoded media on a storage server so another service (something like a content serving thing) knows what storage servers have the media.

Once it has been fully distributed it should also remove the files from GridFS. It should also move the entry from the queue collection to a permanent collection for permanently storing the metadata.

 

Storage servers should also check with each other for new files for faster distribution. This would also fix the problem that the distribution service can't upload media to a storage server that's offline.

 

I'll update the source to GitHub once I have most of the services done or in a state that the full process works.

 

- A 16y/o who loves to code 

[LukeSavenije]