Russian had no open analogues at the time of the development of this project, and we would like other teams to be able to use our work in their projects and promote the topic of creating animations of Russian sign language, since there are many nuances in this area, ranging from transitions between gestures and ending up to the transmission of the correct intonation using gestures.
✅On support.
The project is designed to minimize the need for direct model inference to generate key points and images. Instead, it allows for the extraction of data directly from a pre-built database containing key points and lemmatized words. These data can then be connected using transition algorithms, which in this case operate linearly, selecting the shortest path between consecutive states.
The dev directory is the root, we have modules:
keypoints_creator.pyfor converting videos to dots(Creating a language database)combined_jsons.pyto combine json files of words into a single file for further processingsmooth_transition.pyto create smooth transitions between the last extreme frames (linear transition, requires revision)create_show_frames.pyto create JPEG images and test the resulting skeletonsframes_to_video.pyto create a video from the received frames with a given fps, adjusted to the timeline of the video
update10.mp4
TODO:
- Replace the linear transition between frames with a mathematical model to describe movements according to human anatomy
- Create an interface for adding new items to the project database
- Improve and normalize the markup and dimensions of people in the video
- Dataset "Slovo"
- Mediapipe docs
- Moryossef, A., Müller, M., Göring, A., Jiang, Z., Goldberg, Y., & Ebling, S. (2023). An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation. In Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), June 2023. ZurichNLP/spoken-to-signed-translation Available on: arXiv:2305.17714