PyTroch implementation of Sequence to Sequence model for machine translation, where two recurrent neural networks work together to transform one sequence to another.
Attention mechanism allow decoder decide on which input words to focus during generation of each output word.
In attention matrix a row corresponds to computed attentions over words from input sentence in the moment of generating particular word from output sentence. During generation of first two words 'there' and 'is' network focuses on 'hay' - second word from input sentence instead on 'no'. More examples in the end.
Encoder reads an input sequence and outputs single context vector and hidden states from every timestep.
Decoder recieves context vector and with the help of encoder hidden states produces an output sequence.
Network generally learns to translate unseen sentences from Spanish to English. Sometimes translations are perfect, sometimes mistakes are minor and sometimes output have little in common with input. All things considered, it surpassed my expectations. Trainig was done around 100 000 pairs of spanish-english sentences. Spanish vocabulary size was around 16 000 and English around 8 000 words. Embeddings were attached to inputs both in encoder and decoder.
Perfect translations:
Conoci al rey. - I met the king.
El deporte es salud. - Sport is health.
Ellos tienen mucho dinero. - They have a lot money.
Francia es un pais muy interesante. - France is a very interesting country.
Hoy me levanto temprano en la manana. - I got up early in the morning.
La vida es hermosa. - Life is beautiful.
Me gusta leer libros. - I like to read books.
Mucha gente vive en japon. - A lot of people live in japan.
No me gusta leer libros. - I don't like to read books.
Tu perro y tu gato viven bien. - Your dog and your cat live well.
Minor mistakes:
El baloncesto es popular en america - Let's is popular in america.
El es sabio pero tambien ingenuo. - He is an but but naive.
En invierno esta nevando y lloviendo. - Winter is in in it is raining.
Espana ataco a francia. - We were attacked to france.
Ganamos mucho dinero ayer. - We drank a lot of money yesterday.
La historia le gusta repetir. - The story likes to.
La vida es bella pero tambien pesada. - Life is but also as heavy.
Me gusta correr mucho por la manana. - I like to run a lot of the.
Me gusta leer libros y cantar. - I like reading books and sing.
Rusia es el pais mas grande de europa y del mundo. - Russia is the largest country in europe and in the.
Solia ser mejor en el pasado. - It used to be better at the past.
Serious mistakes:
Ayer monte un carro. - I was a car car.
Cuando visitamos a los padres. - When was the parents parents.
En japon mucha gente vive. - She is many people.
En la ciudad puedes andar en bicicleta y en automovil. - In town can walk a and and.
Estuve en el cine ayer. - I was the the the yesterday yesterday.
More attentions examples:
- perfect translation: No lo hago por dinero. - I don't do it for money.
- minor mistake: Nunca he estado en china. - I've never been to in china.
- serious mistake: En america hay gasolina barata. - There are in the only in there are.
