Context | Listening test setup | Results
Results in detail
These are ranked worst to best. Each encode is available for download if you'd like to compare for yourself. Click on the ODG score to see the log from EAQUAL.
c.layne's "You Dodged a Bullet"
| Encoder | Download | Time to encode | ODG |
|---|---|---|---|
| l3enc 0.99a |  |
132.7s | -3.54 |
| There's incredibly distracting twinkling and graininess over the entire track. ffmpeg's MP3 decoder (like used in Audacity) treats this one like it's skipping constantly, and thus only lasts half the runtime. foobar2000 plays the file without issue. | |||
| Shine 0.1.4 | |
7s | -2.93 |
| Watery, tweeting artifacts are super strong on this one, and there's a hissing, ringing whisper on the hi-hat and vocals in the verses. There's also a "blip" at the very start of the track, meaning the encoder took data somewhere as audio. I had no metadata chunks anywhere in my input WAV, I promise. Shine is far closer sounding to the better- |
|||
| BladeEnc 0.94.2 | |
4s | -2.84 |
| Super noticeable whispering on the vocals, even before the band kicks in. Basically any time the snare kicks, the track warbles. I swear I'm hearing a lot of what I can only describe as "MP3 flutter" if I listen super closely. | |||
| XingMP3 1.5 | |
2s | -1.59 |
| Surprisingly listenable! The swishing is mostly noticeable in the crash cymbals in the choruses, which are panned hard left (hard panning is notoriously difficult to encode). It definitely sounds like a 128kbps MP3, but a not at all offensive one. | |||
| LAME 3.100 | |
4s | -1.34 |
| State of the art for 2017 should damn well come out on top, but not quite, something I'll explain in the comments for LAME 3.90. Fairly close to Xing, but ever so slightly cleaner. The hard panned crashes don't trip it up quite as hard, but the ride cymbal still rings very faintly through the entire chorus. It doesn't sound quite as "full" as Xing? Like Xing is either exaggerating the bottom end of the track or LAME is suppressing it. | |||
| Fraunhofer 3.4.0 | |
2.8s | -1.50 |
| About neck and neck with LAME 3.90! I think the warbling ride in the choruses is just a tiny bit more noticeable here, and the hard-panned crash sounds different from any of the other encoders somehow. | |||
| LAME 3.90 | |
4s | -1.45 |
| Yes, LAME 3.90 came out on top of all the encoders, for this song at least. It somehow sounds cleaner than its cousin from 15 years later! I'm not hearing much artifacting at all on the snare in the verses, while on 3.100's encode, it sounds like the snare is "breathing" slightly. I swear the ride cymbal ringing in the choruses, while still present, is much less noticeable, though the hard-panned crashes definitely warble some. | |||
alaska!'s "Nightmare-X"
| Encoder | Download | Time to encode | ODG |
|---|---|---|---|
| l3enc 0.99a | |
110.8s | -3.39 |
| This one made me laugh. Each pluck of each string becomes a dead, wobbly mess, and the vocals need to be heard to be believed. Literal underwater noises. | |||
| Shine 0.1.4 | |
6s | -2.91 |
| Again with the blip at the start of the track. Funnily enough, both happened in the left ear, suggesting an encoder defect regardless of what you feed it. The tape hiss becomes a bed of watery noise, the guitar picking is definitely smeared, and the entire mix starts to warble when the vocals kick in. Bad/10. | |||
| BladeEnc 0.94.2 | |
5s | -2.26 |
| A little messy on the guitar and tape hiss, but again, the vocals are really what trips this one up. They seem a lot quieter than they should be at first, as if it filtered out all the body in their voices. | |||
| LAME 3.90 | |
2s | -0.76 |
| 3.90 seems to handle the tape hiss the least gracefully of the top encoders. | |||
| Fraunhofer 3.4.0 | |
1.8s | -0.46 |
| Clean encode, and the fastest of the bunch. The piano in the bridge sounds a little smeary, but only very briefly. | |||
| XingMP3 1.5 | |
2s | -0.41 |
| The guitar sounds incredibly clean on this one, very impressive! Even their voices came out pretty clean. Impressively, EAQUAL liked it the best too. | |||
| LAME 3.100 | |
3s | -0.47 |
| About on par with Fraunhofer and Xing. All four of the top encoders in this test are rather interchangeable. | |||
The Dandy Warhols' "Shiny Leather Boots"
| Encoder | Download | Time to encode | ODG |
|---|---|---|---|
| l3enc 0.99a | |
133.3s | -3.73 |
| This one has high-pitched chattering through the entirety of the track, and there's some weird, quick dropouts in places, especially between 20s and 30s. Pretty annoying! Shoutout to that click right at the start. | |||
| BladeEnc 0.94.2 | |
5s | -2.05 |
| Definitely a lot of graininess and sizzle on the high end of the ambience and on the sibilant parts of the vocals. Noticeable, but on the upper end of annoying. Not the worst I've heard from Blade. | |||
| Shine 0.1.4 | |
6s | -1.97 |
| Similar performance to Blade, but I think I give Shine the edge. The vocals feel less smeared and distorted, though it's still pretty audible. | |||
| XingMP3 1.5 | |
2.4s | -1.18 |
| Wow, the openness of the soundstage is palpable coming from the lower-ranking encodes! The artifacts are much less noticeable; they're still there, but they're not bad at all. No complaints at all about how Xing's performed over these songs, given its age. | |||
| Fraunhofer 3.4.0 | |
2.1s | -1.04 |
| Pretty ace, but I do feel some of the vocal echo is less clean and more smeared than the other higher-performing encoders. I wouldn't notice unless I was comparing, though. | |||
| LAME 3.100 | |
5s | -0.83 |
| I'll be honest, both versions of LAME sound pretty close to transparent to me. Perhaps it's my speakers, but even when I listen for the high frequencies or for the vocal echo, I can't notice much distortion at all. A few spots in the track maybe? (~2:45?) | |||
| LAME 3.90 | |
4s | -0.76 |
| See above. I give 3.90 the edge because EAQUAL liked it better, some of the perceived problem spots in the 3.100 encode seem a little better in the 3.90 one, and it finished faster, even if only marginally. | |||
Results in summary
Here's what didn't surprise me: l3enc and Shine performed like garbage. One's old, the other's simple, but it does show how wildly different encoders can perform. Blade performed slightly less like garbage. I've long used Blade to destroy samples of audio for my music before, so I knew how rough it was quality-wise.
Both versions of LAME and the ACM Fraunhofer encoder were very good. Even 20-year-old LAME is still head and shoulders above its competition, and Fraunhofer just had the money to throw at improving their codecs.
What did surprise me was how incredibly well Xing did in every test. The outputs were very clean for its age and for the low bitrates I gave it, and it always came first or runner up in execution speed. I'm not the only one who's found results like this. I think Xing is still very much worth a look if anyone needs an MP3 encoder, especially the newer, more optimized Helix versions of it.
EAQUAL and I have a surprisingly similar set of ears. It's very easy to imagine things you're hearing in a sample of audio, especially one that was not done with a blind control like this one wasn't, so hearing the bot reach similar results was encouraging and curious.
Finally, the sizes of all the encodes didn't vary too much. All encodes of "You Dodged a Bullet" came out to ~4.3MB, "Nightmare-X" to ~3.8MB, and "Shiny Leather Boots" to ~4.1MB.
Wrapping up
MP3s are a wonderful bit of technology I don't think people think enough about, given their ubiquity. The fact that a song can travel around the world in fractions of a second, in good quality, was unthinkable 40 years ago. You can carry entire closets worth of music with you, and even more if you're streaming it over the Internet.
A lot of people associate MP3 with "poor quality", and sure, they can be, but an encode done right can sound as good as a CD, and with the patents expired, software support is near-universal. I find encoder tech and improvements in encoder development to be utterly fascinating, and MP3 really is just the tip of that iceberg.
I'd love to do more pages about the ins and outs of audio compression. Did you know there was an MP1 and MP2 as well? Did you know there's codecs that can pull CD quality out of a single megabyte or two of data by recreating lost audio on the fly? Have you ever heard of Opus, which streams ridiculously fast, stores ridiculously small, and is one half of what Skype used to use for voice chat and one half of a cutting-edge MP3-like codec? Used Discord's voice chat lately? Opus.
This stuff is a deep topic; all signal processing is. More and more as time goes on, I appreciate it better, especially as I use more retrocomputers and MP3 players with only a few gigs of storage (or even less!) to work with.