blackstork-io/fabric

Built-in `rss` data source

traut opened this issue · 0 comments

traut commented

Description

RSS and Atom are popular syndication feed formats. A whole ecosystem of cyber-security blogs distributes their content through the feeds.

Use Case

External feeds are often summarized into short overviews, and Fabric is ideally suited to cover this use case.

Requirements

  • configuration options
    • basic_auth_username - (optional) string attribute
    • basic_auth_password - (optional) string attribute
  • execution options
    • url -- (required) string attribute. An URL to the external RSS or Atom feed

Behaviour

Since RSS and Atom specs differ, the data stream normalizes data into a custom object that looks something like:

{
    "title": "Channel Title",
    "description": "Channel Description",
    "link": "<URL>",
    "pub_date": "<datetime-obj>",
    "items": [
        {
            "guid": "<item-guid>"
            "pub_date": "<datetime-obj>"
            "title": "Item Title",
            "description": "Description",
            "link": "<item-link>",
        }
    ]
}

Additional Information