Bulklift: A Bulk Transcoding Tool for Large Music Collections
It is the year 2019. Most of humanity has switched to streaming their music from an entity known as "The Cloud". A brave group of rebels hold out, clinging valiantly to their own music libraries but struggling ever harder to sync them across multiple devices.
Rebels, I bring a tool to ease your burden. Bulklift.
Bulklift walks your music library looking for .bulklift.yaml
manifests. It follows the instructions within to transcode any music that's in the same directory.
Features:
- ffmpeg for transcoding
- Generate multiple collections of output audio from a single source tree
- Automatically add replaygain tags to output files using the excellent r128gain
- Inheritable, per-directory config system makes it a breeze to setup
- Manifests are just text files stored alongside your music; no database to corrupt or lose
- Toggle which output tree(s) a source album will be transcoded to
- Include/exclude specific tracks based upon filename globbing
- Copies album art files (gif, png, jpg) unmolested to the output directory
- Passthrough
copy
format to copy files without re-encoding - Minimise IO by transcoding all copies of a source within a single ffmpeg run
- Multithreading (4x faster on my Raspberry Pi media server)
Bulklift has approximately zero bells and whistles. It doesn't try to be clever and decisions about metadata are left to the user. It won't download misspellings of your favourite artist's name from CDDB and won't force strange genres from other people's tags upon your filesystem. Never again will Taylor Swift darken your goth folder.
Bulklift is opinionated about the directory structure you keep your music in. One album, one directory, one .bulklift.yaml
with other settings inherited from directories above. It doesn't care about directory names or how many levels there are in your directory tree above the album.
Filenames are simply reused, but with the extension changed to match the output format.
Requirements
- Python 3.10+
apt install ffmpeg
Building
$ python3 -m venv venv
$ . venv/bin/activate # or variant for your shell
$ pip3 install wheel
$ pip3 install -r requirements.txt
Testing
$ sudo apt install sox # used to generate test data
$ python3 -m unittest discover code/
Notes:
- The tests create a fake tree of input media in a tempdir. To save SSD wear and improve performance you may wish to have
/tmp
mounted astmpfs
.
Usage
source_tree > $ echo 'root: true' > .bulklift.yaml # first declare your root
source_tree > $ bulklift edit # add other root config
source_tree > $ cd "Lady Gaga/Born This Way"
Born This Way > $ bulklift edit # add metadata; enable encoding
Born This Way > $ cd ..
source_tree > $ bulklift transcode . # transcode any new targets
Common Operations
Basic Transcoding
$ bulklift transcode /path/to/media/root
Re-Encode Files of One Type
Say you've just updated libopus to a newer, better version. How to re-encode just the .opus files? Simply delete them and allow Bulklift to create replacements...
$ cd /output/tree/root # be sure to do this on the output, not the source!
$ find . -name '*.opus' -delete
$ bulklift transcode /path/to/source/root
Other Useful Operations
- Find all instances of a malformed parameter:
find . -name .bulklift.yaml -exec grep -H 'format:' '{}' ';'
- Edit a manifest:
bulklift edit [path to dir]
. Default is the current directory. If no.bulklift.yaml
exists Bulklift will try to generate a template based on the directory's contents and the manifests of its ancestors. Remember a dir inherits the permissions of its parents so you can delete most of the config.
Configuration
Yaml, nothing more.
The power of bulklift comes from its inheritable manifest system. At any level in your source tree you can create a .bulklift.yaml
manifest overriding options from the previous.
Let's demonstrate with an example. My source music library looks like this:
.
├── .bulklift.yaml
├── __AMBIENT
│ ├── .bulklift.yaml
│ ├── Artist A
| │ ├── .bulklift.yaml
│ │ ├── 2006 First Album [flac]
│ │ │ ├── .bulklift.yaml
│ │ │ ├── 01 - Introduction.flac
│ │ │ ├── 02 - The Second Track.flac
│ │ │ ├── 03 - 3333333.flac
│ │ │ ├── 04 - Getting Bored Now.flac
│ │ │ └── cover.jpg
│ │ └── 2008 Difficult Second Album [mp3]
│ │ ├── .bulklift.yaml
│ │ ├── 01 - Track 1 title.mp3
│ │ ├── 02 - Track 2 title.mp3
│ ├── Artist B
│ │ ├── 2006 First Album [flac]
...
- At the top level set global config options (e.g. path to ffmpeg) and declare target trees with their default bitrates and formats. This must contain
root: true
so Bulklift knows where to stop when loading manifests from subdirectories deep in your tree. - At an intermediate level (
__AMBIENT
) a.bulklift.yaml
manifest specifying metadata for the genre (genre: Ambient
). - At the artist level, a third
.bulklift.yaml
specifies metadata for the artist (artist: Artist B
) - At the bottom level (the individual album) switch encoding on (
enabled: true
) for one or more targets and add metadata tags for the name and year of the album (album: Greatest Hits
/year: 2018
).
The key concept is that Bulklift merges the the manifest of any directory with those of its parents. The deepest manifest (i.e. the one in the dir with your music) has precendence so you can override settings from parent directories. Want to use a different build of ffmpeg to transcode your Dubstep collection? Override the artist for one particular collaboration album? The world is your oyster.
Some of the more common options...
Key | Required | Example | Meaning |
---|---|---|---|
root |
Y | true |
Signifies the root directory of your source tree. Bulklift won't search for any manifests above this. Must be present only in the root manifest; anywhere else and BL will get confused. |
config.transcoding.ffmpeg_path |
- | ${HOME}/.local/bin/ffmpeg |
Ffmpeg binary to use for transcoding. Often this is of value when you want to transcode with a more recent build than the one shipped with your OS. Default is to search your path. |
config.transcoding.threads |
- | 3 |
Number of encoding jobs to run in parallel. Default is the number of available cores. |
config.transcoding.rewrite_metadata |
- | {'track': null, 'album':'', artist':'{artist}'} |
Rewrite selected tags in the target files. Value is treated as a format() string which will have metadata from the Bulklift manifest interpolated into place. An empty value will cause the tag to be deleted. null disables any rewriting inherited from a previous manifest. Valid metadata field names are listed here. |
config.r128gain.r128gain_path |
- | ${HOME}/.local/bin/r128gain |
r128gain binary to use. Default is to search your path. |
config.r128gain.type |
- | album , track , false |
Run r128gain against each target dir after it has been transcoded. Default is album ; other options are track or null (the yaml value, not the string) to disable entirely. |
config.r128gain.threads |
- | 2 |
Run a specific number of r128gain threads. Default is to let it choose, usually the number of cores in your system. |
config.r128gain.ffmpeg_path |
- | ${HOME}/.local/bin/ffmpeg |
Use a specific ffmpeg binary for r128gain. Default is to fail back to config.transcoding.ffmpeg_path and then the first ffmpeg in $PATH . |
config.target.album_dir |
- | "{genre}/{year} {album}" |
Template for the directories music will be transcoded into. The default is suitable for albums with a single artist. Override it for mixes, soundtracks etc. Passed to Python's str.format() method to interpolate metadata fields. Set globally rather than for specific targets because it is presumed you'll want consistent naming. |
outputs |
- | [] |
List of outputs BL may transcode to. While typically (but not necessarily) defined in your root manifest they only take effect for albums in which their enabled flag is set to true . |
outputs[].name |
- | myname |
Textual name for the output. Primarily used for error messages. |
outputs[].enabled |
- | true |
Toggle transcoding for a given output. Default is false and in the normal use case you'll set it to true for any album you want in a given target. NB: Bulklift won't transcode an album unless its directory contains a manifest file, so setting enabled=true at the root level won't have an effect for dirs with no .bulklift.yaml . |
outputs[].sanitize_paths |
- | vfat |
Translate output path to avoid special characters unsupported on vfat/fat32. NB: sanitization is applied to the path generated from config.target.album_dir + the source track name but not to the path to your target tree specified in outputs.<name>.path . |
outputs[].formats |
Y | ['opus', 'mp3'] |
List of codecs the output supports in order of precedence. In the case of the example Bulklift will use existing .opus files if available then fall back to transcoding lossless -> opus, or if that isn't possible using an mp3 file. Typically you'll set this once when defining the output. |
outputs[].opus_bitrate |
- | 128k |
Bitrate to use for libopus. Encoding is VBR so results are approximate. |
outputs[].lame_vbr |
- | 3 |
VBR setting for libmp3lame. Encoding is VBR so results are approximate. |
outputs[].aac_vbr |
- | 3 |
VBR setting for libfdk_aac. Encoding is VBR so results are approximate. |
outputs[].filters.include |
- | ["1-*.flac"] |
List of globs that audio files must match to be included. Applied before any exclude filters. Use a filter like 1* to transcode only the first disc of a two-album set. |
outputs[].filters.exclude |
- | ["*track_i_do_not_like.flac"] |
List of globs audio files must not match to be included. Applied after include filters. |
metadata.* |
Y | - | Mapping of metadata to use for the content. To avoid repetition you can build this up level by level. |
Bulklift will interpolate environment variables used within paths, e.g. ${HOME}/media/target_devices/mp3_player
.
File Naming
Output filenames are copied from the source with the extension changed.
Output directories are a little more complex. I didn't want to rely on the source directory name (mine contain metadata about the format) so the output dir name is freshly generated from the metadata. It defaults to {genre}/{artist}/{year} {album}/
.
Source Formats
Garbage in, garbage out. While Bulklift will merrily transcode from anything your local ffmpeg supports it's best to use a lossless format like FLAC for your sources.
If you can't get lossless audio for your source tree don't worry - a copy
dummy codec is included which will copy the audio into your output tree without transcoding. It is not recommended to transcode from one lossy format to another (e.g. mp3 -> opus) as this results in further loss of quality.
Output Formats
The following codecs can be specified for targets:
Label | Codec | Recommended quality | Notes |
---|---|---|---|
opus |
libopus | libopus | 96k (electronic); 112k (other) |
mp3 |
libmp3lame | 3 (electronic); 2 (other) | The lame encoder producting the venerable mp3 format. Quality levels are for VBR; see their docs. |
copy |
- | - | Copies audio from the source without transcoding. Output will be the exact same bitrate and format as input. Audio bitstream is still run through ffmpeg so we retain the potential to change metadata. Use this if you don't have a lossless copy of the original and don't want to further reduce its quality. |
Tips & Tricks
- Codecs shipped with LTS Linux distributions are often out of date. For those still rapidly improving (e.g. libopus) this is a problem. To transcode with the latest & greatest codecs you may wish to build your own ffmpeg binary. My notes for doing this on Debian are notes/ffmpeg.md. Alternatively you might use one of these static builds.
Why?
Originally I'd rip CD's into mp3. Nowadays disk space is cheap and audio players are good so I've switched to lossless.
Trouble is, not everything has the space (or even support) for lossless audio. Those FLACs need transcoding into other formats to fit on devices. With a handful of devices - each with different space and codec requirements - I found myself spending more time transcoding music than listening to it. Not fun.
Time to automate the process. There are plenty of library management apps already, but all of them embody someone else's idea of how to arrange a music library.
I value:
- Simplicity. It should be easy to understand, easy to extend for new formats and easy to fix when something goes wrong.
- Robustness. Hard to break, easy to fix when it does.
- Transparency. Metadata & settings should be editable with any text editor and stored alongside the music they refer to.
- Configurability. I want to tweak bitrates & formats for parts of my library (e.g. per genre) without having to manually specify them for each album.
- Machine write-ability. Presently I write my
.bulklift.yaml
manifests with vim, but in future I may write CLI tools for bulk-editing the config. - But most of all - never, ever having to do this job again.
License / Contributing
GPLv3 / yes please.