PyAV-Org/PyAV

Unable to set frame rate

tyler-rt opened this issue · 3 comments

tyler-rt commented

Overview

When using a webcam, pyav fails to set frame rate.

Expected behavior

Frame rate should be changeable, as in this ffmpeg script which on my computer works at 60fps:

Actual behavior

Frame rate is not set. cc #1005

Investigation

I made a script with ffmpeg showing that I can change the FPS for my webcam from the default of 30fps to say 60fps or 15fps:

import cv2
import subprocess
import numpy as np
import time

# FFmpeg command
ffmpeg_command = [
    'ffmpeg',
    '-f', 'v4l2',
    '-framerate', '60',
    '-video_size', '1920x1080',
    '-i', '/dev/video4',
    '-f', 'rawvideo',
    '-pix_fmt', 'rgb24',
    'pipe:1'
]

# Start the FFmpeg process and suppress its output
process = subprocess.Popen(ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# Set up OpenCV window
cv2.namedWindow('Webcam', cv2.WINDOW_NORMAL)
cv2.resizeWindow('Webcam', 1920, 1080)

frame_count = 0
start_time = time.time()

while True:
    # Read raw video frame from stdout
    raw_frame = process.stdout.read(1920 * 1080 * 3)
    
    if len(raw_frame) != 1920 * 1080 * 3:
        break

    # Convert raw frame to numpy array
    frame = np.frombuffer(raw_frame, np.uint8).reshape((1080, 1920, 3))

    # Display the frame using OpenCV
    cv2.imshow('Webcam', frame)
    
    frame_count += 1

    # Calculate and print FPS every second
    elapsed_time = time.time() - start_time
    if elapsed_time >= 1.0:
        fps = frame_count / elapsed_time
        print(f"Average FPS: {fps:.2f}")
        frame_count = 0
        start_time = time.time()

    # Exit on ESC key
    if cv2.waitKey(1) & 0xFF == 27:
        break

# Clean up
process.stdout.close()
process.wait()
process.terminate()
cv2.destroyAllWindows()

Reproduction

PyAV is unable to change frame rate:

import av
import cv2
import time
from fractions import Fraction

def open_usb_webcam(max_devices=10):
    "Open VideoCapture with the last valid device (e.g. USB webcam) on the system."
    for device_index in range(max_devices, 0, -1):
        try:
            input_device = f"/dev/video{device_index}"
            # container = av.open(input_device)
            container = av.open(input_device)
            stream = container.streams.video[0]
            return container, stream
        except (av.AVError, IndexError):
            continue
    raise ValueError("No valid video capture devices found.")

# Open the webcam
container = av.open("/dev/video0")
stream = container.streams.video[0]

# Configure the stream for the specified settings
# Tested that we can change the stream width and height
stream.width = 1920
stream.height = 1080
# stream.width = 800
# stream.height = 600
# pix_fmt doesn't appear to matter for setting frame rate
stream.pix_fmt = 'yuv422p'
# however, we are unable to set the frame rate
stream.rate = Fraction(1,60)
# stream.framerate = Fraction(1,15)
# stream.rate = "15"

# Create OpenCV window
cv2.namedWindow('Webcam', cv2.WINDOW_NORMAL)
cv2.resizeWindow('Webcam', 1920, 1080)

# Start time
start_time = time.time()
frame_count = 0

duration = 3
for packet in container.demux(stream):
    current_time = time.time()
    if current_time - start_time >= duration:
        break

    for frame in packet.decode():
        # Convert frame to OpenCV format (BGR)
        img = frame.to_ndarray(format='bgr24')

        # # Display the frame
        # commenting out display has no impact on fps
        cv2.imshow('Webcam', img)
        frame_count += 1

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
        pass

    if current_time - start_time >= duration:
        break

# Calculate and print the average FPS
elapsed_time = time.time() - start_time
average_fps = frame_count / elapsed_time
print(f"Average FPS: {average_fps:.2f}") # 30.00 FPS ???
print(img.shape)
# Release resources
cv2.destroyAllWindows()
container.close()

Versions

  • OS: Ubuntu 22.04
  • PyAV runtime:
PyAV v12.0.0
library configuration: --prefix=/home/reflex/miniconda3/envs/reflex --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1716145014501/_build_env/bin/x86_64-conda-linux-gnu-cc --cxx=/home/conda/feedstock_root/build_artifacts/ffmpeg_1716145014501/_build_env/bin/x86_64-conda-linux-gnu-c++ --nm=/home/conda/feedstock_root/build_artifacts/ffmpeg_1716145014501/_build_env/bin/x86_64-conda-linux-gnu-nm --ar=/home/conda/feedstock_root/build_artifacts/ffmpeg_1716145014501/_build_env/bin/x86_64-conda-linux-gnu-ar --disable-doc --disable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libharfbuzz --enable-libfontconfig --enable-libopenh264 --enable-libdav1d --enable-gnutls --enable-libmp3lame --enable-libvpx --enable-libass --enable-pthreads --enable-vaapi --enable-libopenvino --enable-gpl --enable-libx264 --enable-libx265 --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libopus --pkg-config=/home/conda/feedstock_root/build_artifacts/ffmpeg_1716145014501/_build_env/bin/pkg-config
library license: GPL version 3 or later
libavcodec     60. 31.102
libavdevice    60.  3.100
libavfilter     9. 12.100
libavformat    60. 16.100
libavutil      58. 29.100
libswresample   4. 12.100
libswscale      7.  5.100

Research

I have done the following:

Additional context

WyattBlue commented 
stream.rate = Fraction(1,60)

is deprecated, and is removed in 12.1.0, released today.

WyattBlue commented

anyway, you shouldn't be able to set the stream frame rate? (How would you specify variable frame rates?)

You're looking for either:

  • Changing the device framerate
  • Changing output's framerate by modifying each VideoFrame's pts
tyler-rt commented

@WyattBlue indeed I want to set the device frame rate of the webcam so it acquires at 15 or 60 fps. Is this supported by PyAV currently? In ffmpeg this is done with -framerate 60