Creating a JP to EN Translation dataset from anime subtitles
The aim of this project is to create a translation dataset from Japanese to English from anime subs
The japanese subs can be downloaded from here
The english subs can be downloaded from here
python subs.py --anime_name ANIME_NAME --jp_dir JP_DIR --en_dir EN_DIR
If the subs are simultaneous i.e both japanese and english subs are in the same file, the use the simul_sub.py script
python simul_subs.py --anime_name ANIME_NAME --dir DIRECTORY
The file clean_jp-en_dataset.csv contains JP-EN mapped dataset with 475k samples
The requirement is that the subtitles in the both the directories should be sorted in the ascending or descending order of the episode number.
Right now only .ass and .srt files are supported
Any help on improving the quality of the dataset will be appreciated as i believe there is a lot of room for improvements for my code
The common titles in both the list are shown below, the checked ones are completed
- 07-Ghost
- 5 Centimeter Per Second
- 91 Days
- A.I.C.O._Incarnation
- Accel World
- Acchi Kocchi
- Aggretsuko
- Ai Shite Knight
- Ajin
- Ajin S2
- Akira
- Akarsuki no Yona
- Akuma no Riddle
- Aldnoah Zero
- Amaama to Inazuma
- Amagami SS
- Amagami SS Plus
- Amagi Brilliant Park
- Angel Beats
- Anohana Movie
- Ano Natsu de Matteru
- Another
- Ao no Kanata no Four Rhythm
- Aoi Hana
- Appleseed
- Arakawa Under the Bridge
- Arakawa Under the Bridge x Bridge
- Arslan Senki
- Assassination Classroom
- Baccano!
- Baka to Test to Shoukanjuu
- Bakemono no Ko
- Bakemonogatari
- Bakuman Season 3
- Barakamon
- Beelzebub
- Berserk The Golden Age Arc
- Betterman
- Black Bullet
- Black Jack Final
- Black Lagoon
- Blame
- Blazing Transfer Student
- Bleach
- Boku dake ga Inai Machi
- Bokurano
- Bokura wa Minna Kawaisou
- Boku no Hero Academia
- Bokutachi wa Benkyou ga Dekinai
- Boku wa Tomodachi ga Sukunai
- Boku wa Tomodachi ga Sukunai Next
- Btooom!
- Bungou Stray Dogs S1
- Bungou Stray Dogs S2
- Busou Renkin
- Busou Shoujo Machiavellianism
- Campione!
- Charlotte
- Chihayafuru
- Chio-chan no Tsuugakuro
- Chobits
- Citrus
- City Hunter
- Classroom Crisis
- Claymore
- Code Geass Akito the Exiled
- Code Geass R1
- Code Geass R2
- Cowboy Bebop
- Cross Game
- D-Frag!
- DARLING in the FRANXX
- Danshi Koukousei no Nichijou
- Dungeon ni Deai wo_Motomeru no wa Machigatteiru Darou ka
- Dungeon ni Deai wo Motomeru no wa Machigatte Iru Darou ka Gaiden: Sword Oratoria
- Deadman Wonderland
- Dennou Coil
- Detective Conan
- Devilman Crybaby
- Diabolik Lovers More, Blood
- Digimon Adventure
- Doraemon
- Dororo
- Dr. Stone
- Dragon Ball
- Dragon Ball Super
- Drifters
- Durarara!!
- EVE no Jikan
- Eureka Seven
- Eureka Seven AO
- Fairy Tail
- Fate Apocrypha
- Fate Stay Night
- Fate Zero
- Final Fantasy VII Advent Children
- Free!
- Freedom
- Full Metal Panic Fumoffu
- Full Metal Panic!
- Full Metal Panic! The Second Raid
- Full Metal Panic! Invisible Victory
- Fullmetal Alchemist
- Fullmetal Alchemist Brotherhood
- Fullmetal Alchemist Conqueror of Shamballa
- Gakkou Gurashi!
- Galilei Donna
- Gantz
- Gate Jietai Kanochi nite Kaku Tatakaeri
- Genshiken Nidaime
- Ghost in the Shell (1995)
- Ghost in the Shell The New Movie
- Ghost in the Shell 2: Innocence
- Ghost in the Shell Arise
- Giant Killing
- Ginban Kaleidoscope
- Gintama
- Girls Und Panzer Der Film
- Glass no Hana to Kowasu Sekai
- Goblin Slayer
- Golden Kamuy
- Golgo 13
- Gosick
- Grisaia no Kajitsu
- Guilty Crown
- Hachimitsu to Clover
- Hachimitsu to Clover II
- Hai to Gensou no Grimgar
- Haikyuu S1
- Haikyuu S2
- Haikyuu S3
- Hajime no Ippo
- Hajime no Ippo New Challenger
- Hanasakeru Seishounen
- Hanasaku Iroha
- Hanayamata
- Hanzawa Naoki
- Harmonie
- Harmony
- Hataraku Maou-sama!
- Hayate no Gotoku!!
- Heavy Object
- Hellsing
- Hentai Ouji to Warawanai Neko
- Hibike! Euphonium
- Hibike! Euphonium II
- Hidamari Sketch Hoshimitsu
- Higashi no Eden
- High School Fleet
- High Score Girl
- Higurashi no Naku Koro ni
- Hikaru no Go
- Himouto! Umaru-chan
- Hinomaruzumou - Hinomaru Sumo
- Hourou Musuko
- Hoshi wo Ou Kodomo
- Howls Moving Castle
- Hyouka
- ID-0
- Ichiban Ushiro no Daimaou
- Idoly Pride
- Initial D Fifth Stage
- InuYasha
- Inukami!
- Inuyashiki
- Iriya no Sora, UFO no Natsu
- Isekai Cheat Magician
- Isekai Maou to Shoukan Shoujo no Dorei Majutsu
- Isekai Quartet
- Isshuukan Friends
- Jojo's Bizarre Adventure
- Jojo's Bizarre Adventure Stardust Crusader
- Jojo's Bizarre Adventure Diamond is Unbreakable
- Jojo's Bizarre Adventure Golden Wind
- Jinrui wa Suitai Shimashita
- Joker Game
- Jormungand
- Jujutsu Kaisen
- K
- Kaiba
- Kaiji S1
- Kaiji S2
- Kakegurui
- Kami nomi zo Shiru Sekai
- Kami nomi zo Shiru Sekai S2
- Kami nomi zo Shiru Sekai S3
- Kamisama Hajimemashita
- Kantai Collection
- Kappa no Coo to Natsuyasumi
- Kara no Kyoukai
- Karigurashi no Arrietty
- Katanagatari
- Kemono Friends
- Kemono no Souja Erin
- Keroro Gunsou
- Kiki's Delivery Service
- Kill La Kill
- Kimi to Boku
- Kindaichi Case Files
- Kingdom
- Kishuku Gakkou no Juliet
- Kiznaiver
- Kizumonogatari
- Kobato
- Kobayashi-san Chi no Maid Dragon
- Koe no Katachi
- Kokoro Connect
- Kokoro ga Sakebitagatterunda
- Kono Sekai no Katasumi ni
- Kono Subarashii Sekai ni Shukufuku wo!
- Kotonoha no Niwa
- Koukaku Kidoutai Stand Alone Complex
- Koukaku no Regios
- Kuiba
- Kujira no Kora wa Sajou ni Utau
- Kumo no Mukou, Yakusoku no Basho
- Kuroko no Basket S1
- Kuroko no Basket S3
- Kuromukuro
- Kyoukai No Rinne
- Kyoukai no Kanata
- Little Witch Academia
- Log Horizon S1
- Log Horizon S2
- Long Riders!
- Lost Song
- Lucky Star
- Lupin III
- Lupin III Part III
- Macross Frontier
- Macross Plus
- Made in Abyss
- Magi Labyrinth of Magic
- Magi Kingdom of Magic
- Mahou Shoujo Lyrical Nanoha StrikerS
- Mahoutsukai_no_Yome
- Mahouka Koukou no Rettousei
- Mairimashita! Iruma-kun
- Maison Ikkoku
- Major
- Maoyuu Maou Yuusha
- March Comes in Like a Lion S1
- March Comes in Like a Lion S2
- Marmalade Boy
- Mawaru Penguindrum
- Mazinger Z
- Mikakunin de Shinkoukei
- Mirai no Mirai
- Mitsuboshi Colors
- Mob Psycho 100
- Mob Psycho 100 II
- Mobile Suit Gundam 0083 Stardust Memory
- Mondaiji-tachi ga Isekai kara Kuru Sou Desu yo?
- Monster
- Monogatari Series (TVDB Order)
- Mononoke Hime
- Mouretsu Pirates
- Musaigen no Phantom World
- Mushishi
- Mushibugyo
- Mushishi Zoku Shou
- Nagasarete Airantou
- Nagato Yuki-chan no Shoushitsu
- Nagi no Asukara
- Nande Koko ni Sensei ga!?
- Nanatsu no Taizai
- Naruto
- Naruto Shippuuden
- Natsume Yuujinchou
- Neko no Ongaeshi
- Neon Genesis Evangelion
- Nerawareta Gakuen
- Nichijou
- Nisekoi
- No Game No Life
- Non Non Biyori
- Noragami
- Noragami Aragoto
- Nurarihyon no Mago
- Nyan Koi!
- Occultic;Nine
- Omoide Poro-poro
- One Piece
- One Punch Man
- One Room
- Ookami-san to Shichinin no Nakama-tachi
- Ore no Kanojo to Osananajimi ga Shuraba Sugiru
- Ore no Nounai Sentakushi ga, Gakuen Lovecome o Zenryoku de Jama Shiteiru
- Ouran Koukou Host Club
- Overlord S2
- Overlord S3
- Owari no Seraph
- Owarimonogatari
- Pandora Hearts
- Phi_Brain Kami no Puzzle S1
- Phi_Brain Kami no Puzzle S2
- Phi_Brain Kami no Puzzle S3
- Piano no Mori
- Planetarian ~Chiisana Hoshi no Yume~
- Ping Pong The Animation
- Pokemon
- Pokemon XY
- Pop Team Epic
- Porco Rosso
- Pretty Rhythm Aurora Dream
- Pretty Rhythm Rainbow Live
- Princess Principal
- Prison School
- Psycho-Pass S1
- Psycho-Pass S2
- Psycho-Pass The Movie
- Pupa
- Rainbow days
- ReLIFE
- Rewrite
- ReZero
- Robotics Notes
- Rokudenashi Majutsu Koushi to Akashic Records
- Romeo x Juliet
- Saekano S1
- Saekano S2
- Sailor Moon
- Saint Seiya
- Saiunkoku Monogatari
- Sakasama no Patema
- Sakurako-san no Ashimoto ni wa Shitai ga Umatteiru
- Sakurasou no Pet na Kanojo
- Samurai 7
- Samurai Champloo
- Samurai Flamenco
- Sankarea
- Sarazanmai
- Sayonara Zetsubou Sensei
- School Days
- School Rumble
- Scrapped Princess
- Seikimatsu Occult Gakuin
- Seirei no Moribito
- Seiren
- Seitokai Yakuindomo
- Sekaiichi Hatsukoi S1
- Sekaiichi Hatsukoi S2
- Sekirei
- Sengoku Basara
- Senki Zesshou Symphogear
- Senki Zesshou Symphogear G
- Senki Zesshou Symphogear GX
- Sennen Joyuu
- Serial Experiments Lain
- Servant x Service
- Shakugan no Shana
- Shigatsu wa Kimi no Uso
- Shigofumi
- Shiki
- Shimoneta to Iu Gainen ga Sonzai Shinai Taikutsu na Sekai
- Shingeki no Kyojin
- Shingetsutan Tsukihime
- Shinigami no Ballad
- Shinsekai Yori
- Shisha no Teikoku
- Shirobako
- Shokugeki no Soma S1
- Shokugeki no Soma S2
- Shoujo Kakumei Utena
- Shoujo Shuumatsu Ryokou
- Shouwa Genroku Rakugo Shinjuu S1 & S2
- Shuumatsu no Izetta
- Sirius the Jaeger
- Slayers Next
- Slayers Revolution
- Slayers Try
- Sora no Method
- Sora no Otoshimono
- Sora no Otoshimono Forte
- Sora yori mo Tooi Basho
- Soul land Douluo Dalu
- Space Brothers
- Space Pirate Captain Harlock
- Spirted Away
- Star Driver
- Starry Sky
- Steamboy
- Steins;Gate
- Steins;Gate 0
- Strike Witches
- Suisei no Gargantia
- Summer Wars
- Super Cub
- Suzumiya Haruhi no Yuuutsu
- Sword Art Online
- Sword Art Online Alternative Gun Gale Online
- Sword Art Online II
- Sword Art Online: Alicization
- Tales From Earthsea
- Tales of Zestiria the X
- Tales of the Abyss
- Tamako Love Story
- Tamako Market
- Tanaka-kun wa Itsumo Kedaruge
- Tari Tari
- Tate no Yuusha no Nariagari
- Tensei Shitara Slime Datta Ken
- The Big O
- The_Book of Bantorra
- The Sky Crawlers
- Thermae Romae
- Tiger & Bunny
- Toaru Kagaku no Accelerator
- Toaru Kagaku no Railgun
- Toaru Kagaku no Railgun S
- Toaru Majutsu no Index
- Toaru Majutsu no Index II
- Toaru Majutsu no Index III
- Tokimeki Tonight
- Tokyo Ghoul
- Tokyo Ghoul Root A
- Tokyo Godfathers
- Tokyo Ravens
- Tonari no Seki-kun
- Tonari no Totoro
- Trigun
- Trinity Seven
- Tsubasa Shunraiki
- Tsubasa Tokyo Revelations
- Tsugumomo
- Tsuredure Children
- Tsuritama
- Uchiage Hanabi (2017)
- Uchouten Kazoku
- Uchuu Senkan Yamato
- Uchuu Senkan Yamato 2
- Uchuu Senkan Yamato 2199
- Uchuu Senkan Yamato 3
- Uchuu Show e Youkoso
- Uchuu no Stellvia
- Ultraman Orb
- Ultraman R B
- Urara Meirochou
- Usagi Drop
- Utawarerumono
- Utawarerumono Itsuwari no Kamen
- Vinland Saga
- Violet Evergarden
- Wagaya no Oinari-sama
- Watashi Ga Motete Dou Sunda
- Whisper of the Heart
- Wonder Egg Priority
- World Trigger
- Yakusoku no Neverland
- Yojouhan Shinwa Taikei
- Youjou Senki
- Youkai Watch
- Youkoso Jitsuryoku Shijou Shugi no Kyoushitsu e
- Your Name
- Yu Yu Hakusho
- Yu-Gi-Oh! 5D's
- Yu-Gi-Oh! ARC-V
- Yumekui Merry
- Yuru Camp
- Yuru Yuri S2
- Yuru Yuri San Hai!
- Zero kara Hajimeru Mahou no Sho
- Zero no Tsukaima
- Zetsuen no Tempest
- Zoids Wild
- Zoku Owarimonogatari
- ef - a tale of memories
- Remove/Merge duplicate jp_text to eng_text mappings
- Remove words inside {} and <> in jp_text and en_text
- Remove rows where en_text has less than 3 characters length
- Remove rows with score 0 and number of words greater than 1
- Normalize English translation of duplicate Japanese text i.e same jp text should have same english translation
- Check for Named Entity Tags in the original text and translated text (sanity checks)