NLP to artistic rendering synthesizer.
Sketchr is a natural language processing application running on a Flask web-frontend that takes user inputted sentences, identifies the subjects and associated descriptor words, and synthesizes an image scene.
Sketchr works by using the Spacy natural language processing framework for Python. With this framework in hand, the following steps are taken:
- Text is split into sentences.
- Noun chunks are extracted from the sentence.
- If the noun is new, it is added as a subject to the scene. If the noun is described with reference words (the, it, that, etc.) and the same noun already exists in the scene, then it is merged with the last existing noun.
- The rest of the words in the noun chunk are collected and classified as descriptors
- Rule based matching is used to check for more subject descriptors
- A custom trained NER and location model is used to extract a setting from the sentences for the background
- The inference engine builds, or uses a previously built, statistical n-gram frequency model built using noun chunks from a collection of books to determine more descriptive words to add to the subjects if they aren't descriptive enough.
- A custom built image retrieval engine is used to pull (only valid, non-broken) images from the imagenet dataset.
- The composed scene is sent to an HTML and CSS generator that outputs a static file that is rendered at the "/viewer" endpoint.