FLAC VS MP3? Well, here's my "take it with a grain of salt" answer.

[LeeWonky]

Take this with a grain of salt.


A few days ago, I had an opportunity to work along with a friend who was taking music production in college. I honestly don't know what his thesis is about, but I tagged along since I wanted some of my questions answered. He had all these friends of his who are pretty much audiophiles, or so they say. Basically 10 dudes who know their shit/claim to know their shit. So, we had like 10 people for this and we got all day since we were all in his house.

The song I used is MKAlieZ (Rearranged version) by Sawano Hiroyuki, which he rearranged it himself, (From Aldnoah's OST... which I got after buying the DVD's for this show since I honestly like it and the music is awesome. And goddamnit ScrewAttack, the non-rearranged version is famous because of your Iron Man VS Lex Luthor Death Battle... but oh well) because I think the song introduces alot of elements. Sadly, I can't find the rearranged version of it on YouTube.

So, the song's ripped out of the CD in FLAC and converted the songs to four other MP3 compression rates via Foobar. Note, I did this around a week prior to testing. 

Setup:

So, we did a ghetto ABX testing with the use of two Fiio X3 II's with a Cayin C5 portable amps (he's the reason why we both have the same portable player and amp... cuz he recommended it to me). It had four files of the same song, first (A) as FLAC, second ( B) as 320kbps MP3, third © as 245kbps, fourth (D) with 190kbps, and the fifth (E) with 130kbps (all based on what says on Foobar via conversion). The area we used for testing was my friend's bathroom in his house because it was the most quiet room we could use in the place. We tested two people at a time and only the second half of the song so it'll be faster. And for the sake of their preferred listening volume, we let them adjust it but we use a different song for the sake of making sure they aren't expecting the same song for testing just for adjusting volume. They took their time to listen to the songs as many times as they could, but we were the ones handling the device.

Our question to the dudes: Which copy of the song sounded the clearest?

How they answered: The question/choices were on a piece of paper, they answered via another piece of paper.

Headphone in use for testing: Sennheiser HD600

Rotation was D>A>E>B>C.

Guys who were finished were not told the answer. And for safety, we told them to have fun in my friend's swimming pool (he's rich... yeah).

Please note: Because me and my friend aren't omniscient, one or some of them could've tipped off the rest on which is which when some of them were prolly discussing it by the pool, prolly secretly went inside to tell the rest which are which, so on and so forth.

Results:

Note: Time spent was rounded down, so it could be slightly longer than what I mentioned.
 

Dude #1, #7 and 8: Could not tell the difference and and admitted to guess with A, B, and C, but can tell there's slight degradation with D, and slightly more than that with E.  

Their answers were: #1 with D=190kbps, A=320kbps, E=130kbps, B=FLAC, C=234kbps. Time taken = 20 minutes. #7 with D=190kbps, A=FLAC, E=130kbps, B= 245kbps, C=FLACkbps. Time Taken = 10 minutes.
 

Dude #2 and #10: Knew immediately that A was FLAC, could not differentiate B and C, noticed that D was of lower quality than B and/or C, and says E is of lower quality than D.

His answers were: D=190kbps, A=FLAC, E=130kbps, B=190kbps, C=245kbps. Time taken = 12 minutes.
 

Dude #3-5: Could not tell the difference nor answer with A, B, C, and D.

Their answers were: D, A, B, C=Cannot answer, E=130kbps. Time taken = #3 w/ 10 minutes, #4 w/ 20 minutes, #5 w/ 10 minutes.

Dude #6 and  #8: Got all of them correct.

Their answers were: all correct. Time spent = 20 minutes.
 

Salty conclusion:

 

I think the reason why alot of people think MP3 is a crap compression is probably due to having that much compression actually does screw it up to a degree. But after trying it out myself, by myself, with myself, at home, with the same song, I honestly could not tell the difference in quality other than the 130khz actually making a slight difference that it was enough for me to notice it after 30 minutes worth of listening through the entire song, unlike the test I did with my friend's friends who only had to listen to the second half of the song.

So, does the FLAC vs MP3 debate need to keep on going? Its like RAW vs JPEG. The only difference is, RAW makes sense over JPEG despite being situational to a degree, while FLAC vs MP3 is... well... doesn't matter. All I can say is that I think its placebo effect, or expectation bias. You go with something because you expect it to be or someone else tells you that you should expect it to be, and probably believe that your expectations are met.

 

Also, dude #6 and #8, who produce music and do covers alot, admitted they couldn't tell and had to think hard. #6 said he had a headache doing the test, so that must say something, and #8 said that he thinks it doesn't matter which format would it be for their end product, and would prefer even 192kbps because of the smaller space it takes is easy enough to upload since we live in a country where ISP's are assholes that say, even on the fine print, that there's no throttling unless there's a storm or data cap and shit... but when we reach a certain amount of data, the speed throttles ON A PERFECTLY BEAUTIFUL DAY! So, I understand that.

The test wasn't controlled enough and could use more people for more data, but I only worked with what I was able to work with. So, again, take this with a grain of salt.

[Samfisher]

Just ripping CD's into FLAC doesn't make audio better.  Audio is only better IF the first step, which is recording was recorded in a higher fidelity and incorporates sounds and instruments that show that.  If the recording is bad, it will remain bad regardless of format.  Of course, "bad" is subjective but can be much more obvious in certain types of music and or recording techniques over others.

[Guest]

hey look at that.

 

I still use Vorbis.

[LeeWonky]

Just ripping CD's into FLAC doesn't make audio better.  Audio is only better IF the first step, which is recording was recorded in a higher fidelity and incorporates sounds and instruments that show that.  If the recording is bad, it will remain bad regardless of format.  Of course, "bad" is subjective but can be much more obvious in certain types of music and or recording techniques over others.

I ripped it off of the CD so I can put it on my Fiio X3, then converted the file into four other compressions for the sake of doing this test. I never said that ripping from the CD into FLAC makes it better.

And ofcourse, bad recording will be bad regardless of compression. What I just did in this test is in a sense that "what if the recording was actually good?" kind of scenario. Will compressing it into smaller file sizes degrade it? Turns out, nope. It doesn't do jack shit to its quality either. 128kbps is debatable, IMHO.

[SSL]

One thing that never gets talked about regarding lossy compression is that the algorithms are constantly improving. Things that used to trip up LAME mp3 (e.g. castanets) have been accounted for in later iterations.

[Samfisher]

I ripped it off of the CD so I can put it on my Fiio X3, then converted the file into four other compressions for the sake of doing this test. I never said that ripping from the CD into FLAC makes it better.

And ofcourse, bad recording will be bad regardless of compression. What I just did in this test is in a sense that "what if the recording was actually good?" kind of scenario. Will compressing it into smaller file sizes degrade it? Turns out, nope. It doesn't do jack shit to its quality either. 128kbps is debatable, IMHO.

If there weren't any audio ranges to compress in the recording, then obviously you wouldn't hear a difference It makes the most difference in open stage recordings etc where the dynamic range is so much wider than most audio recordings now for popular music.

[LeeWonky]

One thing that never gets talked about regarding lossy compression is that the algorithms are constantly improving. Things that used to trip up LAME mp3 (e.g. castanets) have been accounted for in later iterations.

Welp, I guess that further cements the reason why we don't really need FLAC as much as before.

 

 

It makes the most difference in open stage recordings etc where the dynamic range is so much wider than most audio recordings now for popular music.

Not sure what you're trying to say on this one, because it took an orchestra to play MKAlieZ's non-rearranged version.

[SSL]

128kbps is debatable, IMHO.

 

That's part of the trouble. For me, 128kbps is awful, but apparently it's only borderline for some people. Of course, the type of music will make a difference.

[LeeWonky]

That's part of the trouble. For me, 128kbps is awful, but apparently it's only borderline for some people. Of course, the type of music will make a difference.

I didn't even have to think about it when it was at 128kbps. I just noticed that it was just off already by that time, even if the effect wasn't really that big on some rap, but it was kind of shit with that song I mentioned, even more with the non-rearranged version. But on some other songs like K-Pop music, its actually rather acceptable and I don't notice the difference with 192kbps.

[Pomfinator]

Not sure what you're trying to say on this one, because it took an orchestra to play MKAlieZ's non-rearranged version.

MKAlieZ does not use a real orchestra, the orchestra is definitely digital, or it was recorded in one of those soundproof rooms with a fraction of the size of a real orchestra.  MkAlieZ is mastered no differently than any modern pop song, aka loud.

 

Though on another note, if we have to specify what kind of song should be used in a lossless vs lossy shootout, we should take a look at our priorities.

[LeeWonky]

MKAlieZ does not use a real orchestra, the orchestra is definitely digital, or it was recorded in one of those soundproof rooms with a fraction of the size of a real orchestra.  MkAlieZ is mastered no differently than any modern pop song, aka loud.

The didn't? Could've sworn it was. But oh well.

[Volbet]

If given a track that I know, I can tell low bttrate (192kbps and below) file from higher bitrate version. However, I can't consistently tell the difference between lossless files and 320kbps MP3's. It is all just guess work on my part.

This do also depend a lot on the recording that I'm listning to. For example, I can't tell the difference between a high and low bitrate version of The Pretty Reckless' first album, becuase it already sounds like they had a 5$ recording budget. 

 

NPR did a similar ABX test some time back: http://www.npr.org/sections/therecord/2015/06/02/411473508/how-well-can-you-hear-audio-quality. 

And people only did a bit better than random guessning: http://www.npr.org/sections/therecord/2015/06/09/412271433/audio-quality-quiz-results-you-did-slightly-better-than-guessing-randomly

But it is fun to read the comments. It's a pretty even mix of people claiming to hear the difference on iPad speakers and people complaining about the music choosen for the test.

 

And as said above, MP3's make way more sense today than they did just 10 years ago. The encoders (LAME in particular) have improved massivly.

Back in the early 2000's I was able to tell different encoders apart just by the way they sounded. Not so today.

[LeeWonky]

If given a track that I know, I can tell low bttrate (192kbps and below) file from higher bitrate version. However, I can't consistently tell the difference between lossless files and 320kbps MP3's. It is all just guess work on my part.

This do also depend a lot on the recording that I'm listning to. For example, I can't tell the difference between a high and low bitrate version of The Pretty Reckless' first album, becuase it already sounds like they had a 5$ recording budget. 

 

NPR did a similar ABX test some time back: http://www.npr.org/sections/therecord/2015/06/02/411473508/how-well-can-you-hear-audio-quality. 

And people only did a bit better than random guessning: http://www.npr.org/sections/therecord/2015/06/09/412271433/audio-quality-quiz-results-you-did-slightly-better-than-guessing-randomly

But it is fun to read the comments. It's a pretty even mix of people claiming to hear the difference on iPad speakers and people complaining about the music choosen for the test.

 

And as said above, MP3's make way more sense today than they did just 10 years ago. The encoders (LAME in particular) have improved massivly.

Back in the early 2000's I was able to tell different encoders apart just by the way they sounded. Not so today.

Yep, like what SSL said, the algorithms have improved greatly since then.  Ofcourse, given that a bad recording will sound bad no matter what, but damn... how great has mp3 become.

[SSL]

Yep, like what SSL said, the algorithms have improved greatly since then.  Ofcourse, given that a bad recording will sound bad no matter what, but damn... how great has mp3 become.

 

Yep, pretty good. The whole point is that it's supposed to be transparent. I seriously do not understand the people who say that the difference between 320kpbs CBR and FLAC is "night and day".

[Dimwitted]

One of the big things is what is going to happen to the files? First gen can be fine but endless copying and reencoding hammers the quality. FLAC is excellent for archiving purposes. Then you get your MP3's. Starting out with a lossly format, even a very good one is just asking for trouble.

[SSL]

One of the big things is what is going to happen to the files? First gen can be fine but endless copying and reencoding hammers the quality.

 

Who does that, though? Do people really burn music CDs anymore?

[spwath]

Who does that, though? Do people really burn music CDs anymore?

Probably only old cd's.

Like I just burned all the good music of our CD collection.

[Rmusic10891]

Who does that, though? Do people really burn music CDs anymore?

Right? I pretty much just use the high bit-rate 320kbps streams from Spotify and call it a day. They, for the most part, sound just fine for me.

[SSL]

Probably only old cd's.

Like I just burned all the good music of our CD collection.

 

Wait you burned it or ripped it?

 

As far as I know, people don't share music on CDs anymore, which seems like the only situation anyone would consider burning and then recopying mp3s.

[PlayStation 2]

hey look at that.

 

I still use ALAC.

ftfy

[spwath]

Wait you burned it or ripped it?

 

As far as I know, people don't share music on CDs anymore, which seems like the only situation anyone would consider burning and then recopying mp3s.

Oh yeah, woops, ripped. Burning is the other way.

 

Yeah, no one burns CD's anymore.

[headnail]

Just to throw my opinion in.....the difference between lossless and lossy codecs becomes more apparent with higher end gear....dac....amp.....speakers/headphones and (some might frown cables.

On mainstream set-ups I find the difference neglible, there is however another factor I believe and that's "accustomization" I guess.

Once you get used to listening on higher end gear you get accustomed to a certain sound, going down be it with lossy codecs or gear you immediately start noticing shortfalls...given enough to time you get accustomed to them as well

[LeeWonky]

Just to throw my opinion in.....the difference between lossless and lossy codecs becomes more apparent with higher end gear....dac....amp.....speakers/headphones and (some might frown cables.

On mainstream set-ups I find the difference neglible, there is however another factor I believe and that's "accustomization" I guess.

Once you get used to listening on higher end gear you get accustomed to a certain sound, going down be it with lossy codecs or gear you immediately start noticing shortfalls...given enough to time you get accustomed to them as well

Paging @SSL 

[SSL]

Paging @SSL 

 

To do what? We all know that cables make no difference and that you don't need "summit-fi" gear to hear the differences in codecs, when audible.

 

Should make an excellent media/gaming center, would also be interesting to see how it fares with an external linear power supply as a dedicated hi-fi music streaming device (with an external multibit tube output DAC of course

 

Obvious troll is obvious?

[LeeWonky]

Obvious troll is obvious?

I did not see that one, my bad.