Use OpenCV to synchronize two videos by determining the needed delay. Enjoy fast performance with multi-threading, which divides video files into chunks for better processing. Enhance your video editing with this efficient solution!
720.mov
During the initial phase, several existing solutions were explored, such as align-videos-by-sound, which uses audio as a track to sync, along with other projects like Sushi, ffsubsync, and AudioAlign.
The primary strategy for syncing two videos consisted of:
- Extracting individual frames as images using ffmpeg
- Selecting a frame from the first video and finding its match in the second video
- Comparing frames using the Structural Similarity Index Measure (SSIM)
Upon realizing that frame extraction was unnecessary, the following changes were implemented:
- Keeping frames in memory as objects, tracking frame number and timestamp
- Introducing threading to process video at multiple timestamps for faster average performance (source)
- Identifying scene transitions and matching them in both videos to reduce false positives
To gain a deeper understanding of video analysis and tools, several resources were consulted:
- Python: Compare Two Images
- Extract timestamps associated with frames ffmpeg extracts from a video
- Extract I-frames to images quickly
- ffmpeg-python
- Video Input with OpenCV and similarity measurement
- Structural similarity (Wikipedia)
- List all possible scene transitions
- Estimate match location in the first video and search around its timestamp in the second video, rather than brute-forcing through the entire video
- Implement speed optimizations and evaluate performance improvements, as suggested by PyImageSearch
- Incorporate audio sync, as mentioned in this StackOverflow thread
- Test alternative video frameworks for speed improvements, such as Decord and VideoGear
- Handle mismatched aspect ratios and potential cropping
- Prioritize search based on winning frame from consecutive SSIM in target SSIM
- Begin brute-force search within a window of the original timestamp, then expand on each failure while avoiding previously searched regions
This project is licensed under the MIT License - see the LICENSE.md file for details