A pipeline that processes documents from a public repository, performs entity extraction + scoring on them and outputs the data into a database in the form of entity-relation graph.
The elements in play in this solution are as follows:
| Element | Description |
|---|---|
| Public Repository | External repository that supplies new documents every day |
| Trigger Web Job | Scheduled to run daily and trigger a flow |
| Query Web Job | Queries for new document IDs (latest) |
| Parser Web Job | Divides documents into sentences and entities |
| Scoring Web Job | Scores sentences and relations |
| External API | API (url) that enables entity extraction and scoring |
| Graph Data | Database to store documents, sentences and relations |
There are 3 web jobs in the bundle
| Web Job | Description |
|---|---|
| Trigger | A schedules web job that triggers a daily check for new document Ids |
| Query | Query documents according to date range provided through Trigger Queue and insert all unprocessed documents to New IDs Queue |
| Parser | Processes each document in New IDs Queue into sentences and entities and pushes them into Scoring Queue |
| Scoring | Scores each sentence in Scoring Queue via the Scoring Service |
To get more information on the message api between the web jobs and the queues see Corpus to Graph Pipeline - Message API
Initiate tests by running:
npm install
npm test
The test replaces the implementation of azure sql database and the azure storage queue with stubs.
In the same way you can replace the implementation of azure sql database and the azure storage queue with non-azure implementations
An exmaple on how to use this project for processing a document in a Genomics context see Corpus to Graph Genomics
Document Processing Pipeline is licensed under the MIT License.