KAHNvert

A distributed media conversion system using ffmpeg and written in Node.js.

This project was inspired by my experience with FileFlows and Unmanic. After many, many hours I was able to get both setup really close to what I wanted, but I could never get either to be perfect.

That's NOT a dig on either project—they're both awesome! I'm just a control freak and didn't want to be limited by number of processing nodes. Plus, I needed a practical project to further my Node.js knowledge and as a portfolio piece.

KAHNvert is a mashup of the word "convert" (as in the noun meaning "someone who has converted") and the emotion of Captain Kirk yelling out "KAAAAAAAAAHN!" in absolute frustration when I couldn't get things working with FileFlows or Unmanic exactly as I wanted. (Again, that's my impatience and neurodivergent need for control—they are both great projects and a million times more mature than this project will likely ever be!)

KAHNvert is built on a client/server model:

The server tracks conversion tasks in a MongoDB database
The clients pull tasks from the server (via REST API) and do the actual conversion (by running ffmpeg locally)

This is a work in progress! I'm currently working on the API. Once it's finished I will implement library scanning, then the actual processing with ffmpeg on client nodes.

How It Works (or will work)

Libraries are setup in the server config
Server scans library folders for files and adds them to the task list
Worker nodes request [n + 1] tasks from server via REST API
- where [n] is the number of preloaded tasks configured
Tasks are then marked as assigned to the requesting node
Worker nodes start copying task files from the library to local temporary directory for processing
Once the file for the first task finishes copying, ffmpeg is run
- files for any additional pre-loaded tasks will continue copying while ffmpeg is running
Output from ffmpeg is sent to server (via REST) as well as logged to stdout on processing node
Status of task is also updated via REST
If the task fails, an error is logged locally and in the task log (via REST)
On task completion (whether successful or not), the next task is loaded and the process repeats until no more tasks are remaining

Retrieving task information

GET /api/v1/tasks - Returns all tasks from database.

Defaults to selecting tasks where status !== "deleted" (as these are considered deleted).
Optional URL parameters:
- libraryId: MongoDB ObjectID; filters tasks by library
- status: String; filters tasks by status
Note: To retrieve tasks regardless of status, set status = "any" in the URL parameters.

GET /api/v1/tasks/:taskId - Returns a specific task.

Defaults to selecting tasks where status !== "deleted" (as these are considered deleted).
To retrieve task regardless of status, set status = "any" in the URL parameters.

GET /api/v1/tasks/:taskId/log - Returns all log entries for task.

More documentation coming.

Creating new tasks

POST /api/v1/tasks/:taskId - Inserts new task.

More documentation coming.

Updating tasks

PUT /api/v1/tasks/:taskId - Updates task.

More documentation coming.

PUT /api/v1/tasks/:taskId/log - Adds log entry for task.

Not supported yet.

Deleting tasks

Not supported yet.

Retrieving library information

GET /api/v1/libraries - Gets all libraries from database.

Not supported yet.

GET /api/v1/libraries/:libraryId/tasks - Gets all tasks for library.

Defaults to selecting tasks where status !== "deleted" (as these are considered deleted).
Not supported yet.

Creating new libraries

Not supported yet.

Updating libraries

Not supported yet.

Deleting libraries

Not supported yet.

Document Schemas

TaskSchema

status: String; One of pending, assigned, processing, complete, missing, failed, or deleted; defaults to "pending"
filename: String; The name of the file to be processed
inputData: VideoInfoSchema; Object containing information about the input video file
outputData: VideoInfoSchema; Object containing information about the output video input file after processing
assignedNode: String; The name of the processing node assigned to the task
startedAt: Date; time when task was started
progress: Number; percentage of progress; defaults to 0
endedAt Date; time when task finished (whether successful or not)
log: Array<TaskLogSchema>; Array of objects containing log information

VideoInfoSchema

streamCount: Number; the number of streams
format: String; the format of the video
formatLongName: String; the long name of the video format
duration: Number; the duration of the video in seconds
bitrate: Number; the bitrate of the video
size: Number; the file size
videoStreams: Array<VideoStreamSchema>; array of video streams
audioStreams: Array<AudioStreamSchema>; array of audio streams

AudioStreamSchema

index Number; original index of the stream
codecName String; audio codec name
codecLongName String; long name of audio codec
profile String; audio stream codec profile
channels Number; number of channels in stream
channelLayout String; channel layout (e.g.: 5.1 or stereo)
sampleRate String; audio stream sample rate
bitrate: String; bitrate of the audio
maxBitrate String; maximum bitrate of the audio
title String; title of the audio stream

VideoStreamSchema

index: Number; original index of the stream
codecName: String; video codec
codecLongName: String; long name of the video codec
profile: String; video profile
pixelFormat: String; video pixel format
framerate: String; video stream framerate
bitrate: String; video stream bitrate
maxBitrate: String; maximum video bitrate
aspectRatio: String; video stream aspect ratio
width: Number; video stream width
height: Number; video stream height
title: String; title of the video stream

TaskLogSchema

timestamp: Date; timestamp of the log entry
message: String; message of the log entry

jesse-kaufman/kahnvert