manojsukhavasi/language-modelling

Exploring different techniques for Language Modelling

Pre-Processing

• Collect the data - ?
• Tokenize the text
• Remove infrequent words
• Prepend special start and end tokens.