Python, File to PNG Compressor.

[WereCatf]

1 minute ago, Poet129 said:

Yes perhaps but it still made it smaller than the original file even if it was png compressed to begin with. This is my point. But I do understand that unless you are going to go out of your way to get potential smaller files depending on the files because 7zip can be better in most instances there is little point in it. But I think that's pretty good for a sixteen year old to figure out... With help.

I haven't seen your code, but have you tried with more complex images? I mean, the picture there only has 164 colours in it -- now I am simply curious to see how it handles one with millions of colours.

[Poet129]

2 minutes ago, WereCatf said:

I haven't seen your code, but have you tried with more complex images? I mean, the picture there only has 164 colours in it -- now I am simply curious to see how it handles one with millions of colours.

If you give me a sample picture I'll do it.

[WereCatf]

2 minutes ago, Poet129 said:

If you give me a sample I'll do it.

This is only 272K colours, but give it a go.

[Poet129]

2 minutes ago, WereCatf said:

This is only 272K colours, but give it a go.

Did you want me to compress this with png compression before as well?

[WereCatf]

2 minutes ago, Poet129 said:

Did you want me to compress this with png compression before as well?

It's already a compressed PNG-file.

[Poet129]

1 minute ago, WereCatf said:

It's already a compressed PNG-file.

735.92KB

739.75KB

742.94KB

LTT-OG.png.7z

[WereCatf]

3 minutes ago, Poet129 said:

739.75KB

Hm, I expected it to make the file even bigger, so I guess that's something.

[Poet129]

Just now, WereCatf said:

Hm, I expected it to make the file even bigger, so I guess that's something.

It did make it bigger than the OG file just not 7zip.

[Poet129]

Also I did a few other files including a video. (The video from last night...) If you want I can give you the results for that too.

[WereCatf]

4 minutes ago, Poet129 said:

Also I did a few other files including a video. (The video from last night...) If you want I can give you the results for that too.

No need to.

[Poet129]

2 minutes ago, WereCatf said:

No need to.

Well I would to thank everybody who participated in helping me make this compressor, helped me test it, and even showed me how it isn't better in most cases. Thanks again, will definitely look for help here again in my future projects.

[WereCatf]

2 minutes ago, Poet129 said:

Well I would to thank everybody who participated in helping me make this compressor, helped me test it, and even showed me how it isn't better in most cases. Thanks again, will definitely look for help here again in my future projects.

If you are interested in compression-techniques and such, you could take a look at how e.g. DEFLATE, the compression PNG-files use, works and try to write your own DEFLATE-compressor. Once you got that working, you could move to more complex ones.

 

https://en.wikipedia.org/wiki/Deflate

[Poet129]

Just now, WereCatf said:

If you are interested in compression-techniques and such, you could take a look at how e.g. DEFLATE, the compression PNG-files use, works and try to write your own DEFLATE-compressor. Once you got that working, you could move to more complex ones.

 

https://en.wikipedia.org/wiki/Deflate

Thanks.

[mariushm]

Like with any compression algorithm,  the compression inside PNG can be configured to trade speed for compression... it's like the presets in 7zip .. fastest,  normal, maximum, ultra

Also, the PNG file format has several filters which can be used to increase compression, and these filters can be applied on regions on the image.

 

A built in (inside a programming language) PNG compressor will never be configured for the highest compression settings, where it takes significantly more memory (ram) and cpu time to compress image.  A built in PNG compressor will almost always be on the "fast" or "normal" compression, and doesn't waste time optimizing the compression - if you want that, you're always free to write your own code or use separate png compression libraries. 

 

So you took a screenshot that's again another special case, pretty much an empty text editor.

A PNG compressor set to maximum compression will look for patterns in the image and take advantage of that and basically compress all that empty space in the image as something like this :  "from 100,50 all the way to 500,500  , there's only 1 color used, rgb(100,50,20) "

A PNG compressor that's configured for using less memory and less cpu time will not search so much in the image, basically as soon as some thresholds are reached it stops because looking in more depth uses a lot more cpu time. So instead of saying only once "100,50 - 500,500 one color rgb(100,50,20)" , a compressor optimized for speed may stop at 32 x 32 or 64x64 tiles and repeat the above 10 times or as many times as it can find 32x32 or 64x64 regions of picture with same color.

 

When you compress the PNG image with 7zip, that's what 7zip finds, it finds those instructions that repeat and compresses them. 

 

So you only manage to compress  the png further with 7zip, because the PNG wasn't compressed with the maximum compression in the first place, and because of limitations of the Deflate algorithm that cause it to miss some things that could be compressed and which 7zip can find.

 

Your last archive contains the same screenshot as previous picture but with color depth reduced to 256 colors . The original picture had around 5400 colors (due to anti-aliasing on the text in VS Code i suspect). Whatever  you used to reduce to 256 colors sucks by the way, seems like it didn't apply dithering.

Anyway...

 

I took the 24 bit screenshot and did a proper 8bit conversion with dithering and the default Irfanview "best" compression (level 9) reduced it down to 18088 bytes.  PNGGauntlet "optimized" the PNG image down to 16259 bytes but spent around 3 seconds on this old laptop to do it... there's other tools which most likely can optimize even better and shrink the image further.... but at the end of the day it's still just ZIP compression.

 

PAQ compressed the original uncompressed image down to 10809 bytes but took almost 58 seconds on this laptop to do it and it's not even the "ultra" preset because -8 needs more ram, but even its "fastest" preset 1 shrinks the original >1 MB image down to 12388 bytes but still takes 7 seconds to do it.

 

7zip bzip2 shrinks the original image down to 14416 bytes, and 7zip ppmd shrinks it down to 14848 bytes, again both better than PNG.  It isn't always the case, with more random stuff in the image it may not compress so well. 

 

Same with the 24 bit color version... saved it as uncompressed (3 MB) and then as png optimized to 43104 bytes.  7zip ppmd was this time better shrinking down to 35102 bytes, probably because bzip2 works with 900 KB chunks, so it's compressing 4 separate regions of the file, so missing some things that repeat between chunks. 

But if you have the time, paq8 shrinks it down to 32007 bytes for the fastest preset, taking 22s to complete it... or 29512 bytes at highest compression setting, but taking 176 seconds to do it. 

Do you expect to press Print Screen and wait 3 minutes for a screenshot to be saved? It's obviously not feasible, there has to be some tradeoffs between compression speed and disk space.

 

c:\Temp\topng>paq8jc.exe -7 picture.8.0.png
Creating archive picture.8.0.png.paq8jc with 1 file(s)...
picture.8.0.png 1052400 -> 10809
1052400 -> 10847
Time 57.59 sec, used 1017681291 bytes of memory

c:\Temp\topng>paq8jc.exe -1 picture.8.0.png
Creating archive picture.8.0.png.paq8jc with 1 file(s)...
picture.8.0.png 1052400 -> 12388
1052400 -> 12426
Time 7.46 sec, used 25886827 bytes of memory

c:\Temp\topng>paq8jc.exe -7 picture.24.0.png
Creating archive picture.24.0.png.paq8jc with 1 file(s)...
picture.24.0.png 3153188 -> 29473
3153188 -> 29512
Time 176.13 sec, used 1017681289 bytes of memory

c:\Temp\topng>paq8jc.exe -1 picture.24.0.png
Creating archive picture.24.0.png.paq8jc with 1 file(s)...
picture.24.0.png 3153188 -> 31968
3153188 -> 32007
Time 22.17 sec, used 25886825 bytes of memory

 

[Poet129]

On 12/30/2020 at 12:29 AM, WereCatf said:

It's already a compressed PNG-file.

Hey I found out PNG compression was working incorrectly.

735.92KB

686.89KB

742.93KB

LTT.png.7z

[mariushm]

Sigh... let's see, the original image is 1000 x 625 pixels , 24 bit per pixel and there's 211755 unique colors so you have to use 24 bits per pixel... so this means you have a total of 1000x625x24 =  15,000,000 bits.

 

Your black and white image is 2376x2361, which has 5,609,736 pixels ... so I can only guess that maybe you reduced the image to 256 colors? That would reduce the original image to 5,000,000 bits but then you'd use a better resolution like 2240 x 2237

 

Did you just convert the compressed png image directly to binary? In that case you screwed up somewhere... the original was 753587 bytes or 6,028,696 bits , your png image has 5609736 bits so again it doesn't have all the bits of original picture.

 

Do everyone here a favor... every time you post somethings, post the input, the compressed input and the code that decompresses the compressed input back to the original input ... and before you post, do a file compare between your original input and the decompressed file.  If they don't match, don't post.

 

[Kilrah]

18 minutes ago, mariushm said:

 and before you post, do a file comdon'tpare between your original input and the decompressed file.  If they don't match, don't post.

 

Your "don't" slipped by a few bytes 

[Poet129]

20 minutes ago, mariushm said:

Sigh... let's see, the original image is 1000 x 625 pixels , 24 bit per pixel and there's 211755 unique colors so you have to use 24 bits per pixel... so this means you have a total of 1000x625x24 =  15,000,000 bits.

 

Your black and white image is 2376x2361, which has 5,609,736 pixels ... so I can only guess that maybe you reduced the image to 256 colors? That would reduce the original image to 5,000,000 bits but then you'd use a better resolution like 2240 x 2237

 

Did you just convert the compressed png image directly to binary? In that case you screwed up somewhere... the original was 753587 bytes or 6,028,696 bits , your png image has 5609736 bits so again it doesn't have all the bits of original picture.

 

Do everyone here a favor... every time you post somethings, post the input, the compressed input and the code that decompresses the compressed input back to the original input ... and before you post, do a file comdon'tpare between your original input and the decompressed file.  If they match, don't post.

 

 

2 minutes ago, Kilrah said:

Your "don't" slipped by a few bytes 

New Video

[mariushm]

32 minutes ago, Poet129 said:

 

New Video

1:54

Your "compressor" doesn't treat the input as binary data, as "anything",  you have a special case for PNG images where you use pngcrush to "optimize" the compressed content inside the original image ... this results in a BINARY DIFFERENT PNG image than the original file. pngcrush manages to reduce the image from 753587 bytes down to 700941 bytes by using various strategies on chunks of the image before applying Deflate algorithm to compress the bytes.

 

It is same visual content, but the decompressed file is BINARY DIFFERENT.  Your decompressor MUST produce the same original image, same number of bytes, must be binary exact.

 

Screenshot from your video.

 

 

There are some tools which are used as pre-filters to compressors, like precomp : http://schnaader.info/precomp.php#d

The tool analizes a binary file (doesn't care about extension or anything, just analizes the stream of bytes) and tries to determine if there's chunks of data inside which is compressed using Deflate algorithm (or other algorithms or compression methods) and then figures out the exact parameters that were used by a compressor to create that sequence of compressed bytes. So then the tool can extract that sequence, uncompress it, compress the raw uncompressed stuff with something stronger, and also includes in the file the parameters needed to reconstruct the original sequence of bytes (compress with deflate with those exact parameters to get binary exact file)

 

c:\Temp\precomp\windows>precomp.exe -v -cn LTT.png

Precomp v0.4.7 Windows 64-bit - ALPHA version - USE FOR TESTING ONLY
Free for non-commercial use - Copyright 2006-2019 by Christian Schneider
  preflate v0.3.5 support - Copyright 2018 by Dirk Steinke

Input file: LTT.png
Output file: LTT.pcf

Using packJPG for JPG recompression, packMP3 for MP3 recompression.
--> packJPG library v2.5k (01/22/2016) by Matthias Stirner / Se <--
--> packMP3 library v1.0g (01/22/2016) by Matthias Stirner <--
More about packJPG and packMP3 here: http://www.matthiasstirner.com

(0.01%) Possible zLib-Stream in multiPNG found at position 37
Compressed size: 753260
Can be decompressed to 2500625 bytes
Non-ZLIB reconstruction data size: 43484 bytes
New size: 2544358 instead of 753587

Done.
Time: 1 second(s), 123 millisecond(s)

Recompressed streams: 1/1
PNG streams (multi): 1/1

So the tool  parsed the file and detected a Deflate compressed sequence and decompressed it and created a 2,500,625 bytes file that also contains the parameters needed to re-create the original file, binary exact.

 

Now I can take this 2.5 MB file and compress it with whatever compressor I chose :

 

7zip ppmd : 615470 bytes

7zip bz2 :  673344 bytes

7zip lzma : 669586 bytes

 

paq8jc best (-7):  537102 bytes in 169.5s

paq8jc fastest (-1) : 568509 bytes in 21s

 

So now I can uncompress the LTT.pcf ("filtered" file) file and get back my 2.5 MB file and then i can run pre-comp again on that and get back my binary exact image :

 

c:\Temp\precomp\windows>precomp.exe -v -r -oLTT2.png LTT.pcf

Precomp v0.4.7 Windows 64-bit - ALPHA version - USE FOR TESTING ONLY
Free for non-commercial use - Copyright 2006-2019 by Christian Schneider
  preflate v0.3.5 support - Copyright 2018 by Dirk Steinke

Input file: LTT.pcf
Output file: LTT2.png

Using packJPG for JPG recompression, packMP3 for MP3 recompression.
--> packJPG library v2.5k (01/22/2016) by Matthias Stirner / Se <--
--> packMP3 library v1.0g (01/22/2016) by Matthias Stirner <--
More about packJPG and packMP3 here: http://www.matthiasstirner.com

Uncompressed data, length=37
Decompressed data - PNG multi
Header length: 2
Reconstruction data size: 43484
Recompressed length: 753260 - decompressed length: 2500625
Uncompressed data, length=20

Done.
Time: 655 millisecond(s)

 

Now if I want to, I can run file compare on both files and be sure they're bit exact : 

 

c:\Temp\precomp\windows>fc /B LTT.png LTT2.png
Comparing files LTT.png and LTT2.PNG
FC: no differences encountered

 

I'm attaching precomp below with the compressed files as proof and maybe for you to learn something by doing your own examples.

proof.7z

[Poet129]

7 minutes ago, mariushm said:

It is same visual content, but the decompressed file is BINARY DIFFERENT.  Your decompressor MUST produce the same original image, same number of bytes, must be binary exact.

I was aware of this however I saw no reason to undo the pngcrush. However to please you I will find a way.

[mariushm]

1 minute ago, Poet129 said:

I was aware of this however I saw no reason to undo the pngcrush. However to please you I will find a way.

The reason is the file is not binary identical. 

A lossless compressor-decompressor  MUST produce the same file when decompressed.

If you're fine with lossy compression, then you may just as well save that LTT.png image as JPG with 99% image quality, and visually would look the same, but use less space. 

 

By the way, also keep in mind that PNG images can contain extra data besides the raw image, it just happened the image you chose doesn't contain anything else.

As an example, see EXIF data or IPTC info ... see the image below I took some time ago, which contains EXIF data that includes even the GPS location of where the picture was taken (i blurred it because I don't particularly care for people to know where I took the picture) ... but the point is that PNGcrush may or may not throw out this EXIF data because it's not needed to see the image on the screen. But .... I may care about it and want to keep it, to catalogue a collection of images or whatever.

 

The PNG specification also allows for TEXT COMMENTS and lots of optional tags - see 4.7 and 5.4 in the specification: https://www.w3.org/TR/2003/REC-PNG-20031110/#4Concepts.Format

Just because it's visually not relevant, doesn't mean it can be lost.

 

[Poet129]

10 minutes ago, mariushm said:

The reason is the file is not binary identical. 

A lossless compressor-decompressor  MUST produce the same file when decompressed.

If you're fine with lossy compression, then you may just as well save that LTT.png image as JPG with 99% image quality, and visually would look the same, but use less space. 

Just because it's visually not relevant, doesn't mean it can be lost.

It may not be binary identical but it could be converted back, read first two sentences from site.

[WereCatf]

2 hours ago, Poet129 said:

It may not be binary identical but it could be converted back, read first two sentences from site.

Well, no. Pngcrush has no way of getting back data that was removed, like e.g. if you remove colour-profiles, gamma-details etc.

[Poet129]

7 hours ago, WereCatf said:

Well, no. Pngcrush has no way of getting back data that was removed, like e.g. if you remove colour-profiles, gamma-details etc.

I've configured it to only clean up the "junk" repeated data in the file not data that is potentially important.

[Sauron]

15 hours ago, Poet129 said:

I've configured it to only clean up the "junk" repeated data in the file not data that is potentially important.

This is not what lossless compression is though... you need to get back the exact file you put in. And it should work with files that aren't a PNG image, too. Plus it doesn't really make any sense to use it if it results in an image that is larger than the original file.

 

If I just wanted to reduce the size of a PNG image without visual quality downgrades I could just open gimp and export it in PNG with maximum compression, removing "unimportant" data such as the thumbnail.

 

 

This results in a visually identical image that is "only" 714Kb in size. Too bad I lost all the metadata which I'm never getting back. The image looks exactly the same though:

 

 

and it's smaller than your script's output without requiring another script to get something resembling the original image. Perhaps more importantly, it's not larger than the original image.

 

The truth of the matter is that PNG already has excellent lossless compression baked in and therefore any attempts to losslessly compress it further will result in a file that is the same size or larger, as seen with 7zip which also adds some metadata to the archive itself - unless you can put in the academic work required to potentially find a better algorithm than what PNG uses.