Stream video as series of frames (image buffers)

[benwiley4000]

I think my problem might be slightly similar to #1 but I want to stream a video and get each frame as a buffer, as it becomes available.

Is this possible without the "frame" event mentioned in the todo list?

[phaux]

You have to set output format to mjpeg and then split the stream manually by looking at the bytes. You can implement a transform stream which does this:

const {Transform} = require('stream')
class ExtractFrames extends Transform {
  constructor(delimiter) {
    super({readableObjectMode: true})
    this.delim = Buffer.from(delimiter, 'hex')
    this.buffer = Buffer.alloc(0)
  }
  _transform(data, enc, cb) {
    // Add new data to buffer
    this.buffer = Buffer.concat([this.buffer, data])
    while (true) {
      const start = this.buffer.indexOf(this.delim)
      if (start < 0) break // there's no frame data at all
      const end = this.buffer.indexOf(this.delim, start + this.delim.length)
      if (end < 0) break // we haven't got the whole frame yet
      this.push(this.buffer.slice(start, end)) // emit a frame
      this.buffer = this.buffer.slice(end) // remove frame data from buffer
      if (start > 0) console.error(`Discarded ${start} bytes of invalid data`)
    }
    cb()
  }
}

And then use it like that:

const conv = ffmpeg()
conv.output({f: 'image2pipe', vcodec: 'mjpeg'})
  .pipe(new ExtractFrames('FFD8FF')) // use jpg magic number as delimiter
  .on('data', frame => /* do things with frame */)
  .on('end', () => /* do things on complete */)
  .on('error', () => /* do things on error */)

[phaux]

I added this question to the FAQ in README. Closing.

[oste]

@phaux I tested this solution with the attached matroska file and it is only outputting 738 frames instead of 739 that you can see in the debug output

ffmpeg-stream log: frame= 739 fps=108 q=24.8 Lsize= 18109kB time=00:00:29.56 bitrate=5018.7kbits/s speed=4.33x +0ms

Do you have any ideas what might be causing this?

test.mkv.zip

[phaux]

@oste You would need to add the _flush method to ExtractFrames, which emits the last frame. _transform only emits a frame when it sees the beginning of the next frame so it will always skip the last one.