Automatic pause on scene change when generating the script
ilyanikolaevsky opened this issue · 9 comments
Current solution with asking the user to press 'q' when the scene changes is suboptimal.
Main source of pain during automatic funscripting is if you miss the scene change, then the tracking algorithm would detect some random position in a new scene, which would mess up the range of the previous points.
It would be great if the automatic tracking UI had a pause/unpause functionality and then on each detected scene change the user could either press 'q' and set up the range or press 'space' and continue the tracking on a false-positive scene change detection.
It might be easier to have automatic tracking pause when it hits an existing point. That way, you could "pre-cue" the scene changes by going through the video and dropping in points at the "correct" values.
Do you have a suggestion how i can automatically detect such a scene change? I have no idea how to automatically detect this without too much computational effort. You can of course apply a 3d pose estimation with deep learning but this would probably slow down the tracking 100-1000 times.
Currently there is one new function that allow to specify a area where you expect the tracking prediction and if the prediction is outside this specified area the tracking stops (see this commit) - This feature will be available in the next release.
I think the "define start and end points" feature should do the job nicely.
I meant "scene change" as in video codec context. If the current frame differs from the previous too much - there's a high chance that the tracked region is invalid now. I don't know what facilities OpenCV has for that, but it sure should have something. If the big chunk of the frame doesn't have any valid motion vectors, or simply too many pixels have changed - that should trigger a pause. Maybe some threshold values should be exposed in config.
An easy hack would be to pause when the tracked region has jumped suddenly (too big of a change from the previous frame, or position outside of the usual range - e.g. what was during the first 5 seconds of the processed video chunk). It's not an ideal solution, but it would remove the main issue, caused by the change - the misidentified region messing with the range selection at the end of tracking.
Some googling returned this: https://github.com/Breakthrough/PySceneDetect
So, it's possible to detect scene changes with openCV. Maybe it's feasible to use the same openCV API here to automatically detect pause locations.
Or, you could invoke this packet to find the pause locations.
Or implement a list of predetermined pause locations, which the users could generate themselves using e.g. the packet above.
Thanks for the link. I will have a look at the python module in the next days.
@michael-mueller-git, thank you!
The Feature is now available in the new release v0.1.1. With default setting the application read the csv from PySceneDetect in the same directory where the video is located. If such a csv file not exist this feature get automatically disabled. You can also use an build in online scene change algorithm by change this config entry. The config files for MTFG with Install Option 1 are located in %appdata%/OFS_data/extensions/Funscript Generator Windows/funscript-editor/funscript_editor/config
It would be helpful if you could try out the implemented online methods and give feedback if they work reliably and detect few false positive scene changes. Then we could set one of the built-in methods as default value in a future release.
Thank you very much!