Python preferably, Network Compression?

[Poet129]

I made a compressor that is particularly good with small files, so I finally thought of a use case I compressed a 1526 Byte file down to 537 Bytes. How would I go about making this work as network compression?

[Electronics Wizardy]

How do you wnt to compression network traffic? Are you in control of both ends?

 

Use the compression for something like a vpn?

[Poet129]

3 minutes ago, Electronics Wizardy said:

How do you wnt to compression network traffic? Are you in control of both ends?

 

Use the compression for something like a vpn?

I would be in control on both ends. Yes, I'm thinking a slow internet connection to a fast internet connection then with compression vpn in between.

[Electronics Wizardy]

7 minutes ago, Poet129 said:

I would be in control on both ends. Yes, I'm thinking a slow internet connection to a fast internet connection then with compression vpn in between.

Well most vpns support this already, but the problem is a lot of traffic is already encrypted or compressed, so you can't compress much more.

[Poet129]

1 minute ago, Electronics Wizardy said:

Well most vpns support this already, but the problem is a lot of traffic is already encrypted or compressed, so you can't compress much more.

Any that are open source?

[Electronics Wizardy]

Just now, Poet129 said:

Any that are open source?

open vpn can do this.

[Poet129]

Just now, Electronics Wizardy said:

open vpn can do this.

Okay thanks.

[Eigenvektor]

4 hours ago, Poet129 said:

How would I go about making this work as network compression?

For network compression, the compression factor is typically less important than the algorithm's speed. Having the best compression is useless if it results in slower transfer speeds (due to the extra time needed to compress/decompress). It also needs to be able to compress data on the fly (while it is being streamed), as opposed to compressing it "offline" and then sending the compressed file.

 

As @Electronics Wizardy said and I pointed out in your previous question about network compression, most content provided by web servers is compressed already. You cannot compress it further. In most cases attempting to compress data that is already compressed will even increase its size.

 

If, for example, you look at the network headers from LTT, their web server is using Brotli compression (content-encoding: br). If you look at Wikipedia's header instead you will see "gzip" which is another compressor. So the content delivered by either web server is compressed already. The same is true for most other web servers.

 

Last but not least, compressing VPN traffic is not recommended for security reasons, because it can result in leaked secrets. In any case, OpenVPN already supports compression using lzo.

[mariushm]

@Poet129you didn't make a compressor that's particularly good at small files, you're just converting files to PNG images which use deflate to compress the content. Cut this crap and confusion.  Skip the png generation and just use deflate directly. 

As several of us explained, you managed to just use some bad inputs (more compressible than usual) to get some reduced file sizes (but you would have gotten that with regular compression anyway) and now you think you have something. You don't.

 

[DevBlox]

Next up - using jpeg for network compression.

Spoiler

Hint for the 'joke' - it's a lossy compression type.

 

On a more serous note - deflate will not be quick enough network traffic. It's not a good idea. Yes, it yields a better compression ratio, but it's relatively slow speed will *insert a more violent/expressive phrase for 'utterly destroy'* the performance of a network. That's based on compression benchmarks on larger files (can't find any other sources at the moment), so what you will see does depend on a lot of factors. If there's any seriousness to this - it must be thoroughly benchmarked. That might be: compression ratios, added latency, CPU usage, memory usage. OpenVPN is already not doing too well on the CPU usage (sometimes latency too) part.

 

I've not heard any prominent projects using deflate for network traffic, I either don't know enough projects, or the reason stated above is in play, and no one's willing to use it because they've done their research.

 

Be careful when taking advice from the OpenVPN codebase though - it's already a heap in some significant ways (it's single-threaded and user-space only for a start). Things done there are not in line with high performance VPN networking for the most part.