- FLAC (flac v1.3.2): Downloaded from https://xiph.org/flac/download.html
- MP3 (lame 3.100.1): Downloaded from https://www.rarewares.org/mp3-lame-bundle.php
- APE (mac 5.69): Downloaded from https://www.monkeysaudio.com/download.html
- Wavpack (wavpack 5.3.0): Downloaded from http://www.wavpack.com/downloads.html
- TAK (takc 2.3.0): Downloaded from http://thbeck.de/Tak/Tak.html
- TTA (tta 2.3): Downloaded from https://sourceforge.net/projects/tta/files/tta/ttaenc-win/
- WMA (WmaEncoder 0.2.9c): Downloaded from https://hydrogenaud.io/index.php/topic,90519.0.html
- lossyWAV (lossyWAV 1.4.2): Downloaded from https://wiki.hydrogenaud.io/index.php?title=LossyWAV
- lossyFLAC = lossyWAV + FLAC; lossyTAK = lossyWAV + TAK; lossyWV = lossyWAV + Wavpack
- ALAC (refalac v2.71): Downloaded from https://github.com/nu774/qaac/releases
- OptimFrog (ofc & ofs v5.100): Downloaded from http://losslessaudio.org/Downloads.php
- Opus (opusenc & opusdec 1.3): Downloaded from https://opus-codec.org/downloads/
- Ogg Vorbis (oggenc 2.88): Downloaded from https://www.rarewares.org/ogg-oggenc.php
- AAC
- (qaac v2.71): Downloaded from https://github.com/nu774/qaac/releases, (follow instructions here)
- (NeroAac v1.5.1): Downloaded from http://wiki.hydrogenaud.io/index.php?title=Nero_AAC
- Musepack (mpcenc 1.30.0 & mpcdec 1.0.0): Downloaded from https://musepack.net/index.php?pg=win
For missing encoder or decoder, ffmpeg (4.3.1-2020-11-19 x64)
is used. Also note that OptimFrog encoding and decoding always crashes for me, and the dual stream encoding cannot preserve lossless audio (due to the crash , I guess).
The script can run regardless of your OS, but the preset codecs parameters and included binaries are for Windows. You will need Python 3 and following libraries installed to run the benchmark.
numpy
scipy
librosa
matplotlib
In order to run the benchmark, you have to
- Change the audio sources in the
test.py
- Collect missing dlls for
qaac
as mentioned above - Then execute
python test.py
There are several things you can customize in test.py
- Change codecs parameters by change
lossless_codecs
andlossy_codecs
dictionaries. - Choose x86 or x64 version of the codecs by switch
x64
variable. - Add your test audio files in
files
list.
- Compression Ratio (CR): compressed file size / original uncompressed file size (lower the better)
- Average Bitrate: compression ratio times original bitrate
- Spectrogram Error: direct difference between power spectrograms (lower the better)
- Weighted Spectrogram Error: direct difference between A-weighted power spectrograms (lower the better)
- Signal-to-Noise Ratio (SNR): noise is calculated from signal power difference (higher the better)
- Weighted SNR: noise is calculated from A-weighted signal power difference (higher the better)
When calculating the compressed file size for hybrid lossless audio codec (like wavpack -c -b192
), the size of both compressed audio and correction file are counted in lossless charts, while only the size of compressed audio is counted in lossy charts.
Note that not all results are shown in the plots since they will make the figure looks messy. Only best results in a series of codecs are plotted. For example in encoding time - average bitrate figure, only left-bottom data points are shown for an audio codec. For codec parameters in the figure, there's a Full Charts section below figures for each audio file. For complete results with numbers, please refer to result.json or result.md
- Audio Spectrogram
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison
Full Charts
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison
- Audio Spectrogram
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison
Full Charts
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison
- Audio Spectrogram
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison
Full Charts
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison
- Audio Spectrogram
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison
Full Charts
- Lossless Codecs Performance Comparison
- Lossy Codecs Performance Comparison
- Lossy Codecs Quality Comparison