fast-crawler/fastcrawler

Abstraction layer in Spider

Opened this issue · 0 comments

ManiMozaffar commented
  • Define new abstraction, so the process would be unaware of what component is Spider
  • In result, we should be able to add Parser, Saver and etc
  • The abstraction layer, may cache data

Final Result
Process(Spider() >> Parse(model=ParsingModel) >> Save(to=PostgreSQL)