Vehnem/ld-crawl

Linked Data Crawl and Retrieval

Linked Data Crawl and Retrieval

• ld.core - core components
• ld.crawler - linked data crawler service

Architecture

KnownIRIs

QUEUE

Crawler

Metadata Storage

**

Backlog

• add https://blog.worldline.tech/2020/04/10/split-unit-and-integration-tests.html
• add loader class

Load Balancer

Use Bucket system to distribute domain workload

Accessibility Issues : Crawler Errors and Exceptions

Issues of Linked Data Impl

• Content Negotiation is to difficult [Aidan Hogan]

Blocking

is there a way around the thread.sleep() for next group function