BGPKIT Broker Backend is the backend API and data collection implementation for the BGPKIT Broker project.
Users can use this repository to deploy a BGPKIT Broker instance on their own equipment, and it is free under MIT license! See more in Deployment and LICENSE sections.
At a high level, the BGPKIT Broker is a REST API service that provides search capability for BGP MRT archive files from various data collection projects. An user can search for matching archive files with simple REST API calls.
For example, we want to learn about exactly what BGP updates were involved that caused Facebook’s outage for multiple hours on Oct. 4th, 2021. We can use the following REST call to the BGPKIT Broker API to gather the full list of data files that may contain the relevant updates for the event:
GET https://api.broker.bgpkit.com/v2/search?ts_start=1633362000&ts_end=1633363200&data_type=update&page_size=5&page=1
{
"count": 5,
"page": 1,
"page_size": 5,
"error": null,
"data": [
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views2",
"data_type": "update",
"url": "http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 1153433,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views3",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views3/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 8388608,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views5",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views5/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 22528,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views.amsix",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views.amsix/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 7864320,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views.chicago",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views.chicago/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 1887436,
"exact_size": 0
}
]
}
// GET https://api.broker.bgpkit.com/v2/search?ts_start=1633362000&ts_end=1633363200&data_type=update&page_size=5&page=1
// HTTP/1.1 200 OK
// Content-Length: 1395
// Content-Type: application/json
// Date: Mon, 25 Apr 2022 21:54:29 GMT
// Server: Caddy
// Server: uvicorn
// Request duration: 1.434210sHere is a break-down of the call parameters:
ts_start=1633362000: all BGP files after 15:40 UTC on that dayts_end=1633362022: all BGP files before 16:00 UTC on that daydata_type=update: only return updates filespage_size=5: return 5 items per pagepage=1: return the 1st page (pages starting from 1)
For more examples and API spec, please checkout our documentation site.
Broker backend currently support two data collection projects: RouteViews and RIPE RIS. Specifically, we have designed tailored data collection procedures to efficiently crawl the data publication sites from the two projects and collect basic information of the published BGP MRT files.
The information we currently collect for each data file includes:
- data type: either BGP updates files, or BGP table dump files.
- timestamp: the time of the beginning of the data dump, usually aligns on the full five-minute marks.
- url: the URL for the downloadable data file.
- collector ID: the ID of the data collector where that data comes from.
- rought and exact file sizes
Note that the data dump frequencies for RouteViews and RIPE RIS are different. Here is a table summarizing their frequencies:
| Project | Update frequency | Table dump frequency |
|---|---|---|
| RouteViews | one per 15 minutes | one per 2 hours |
| RIPE RIS | one per 5 minutes | one per 8 hours |
In essence, you would see more frequent BGP updates dumps from RIPE RIS and more frequent full table dumps from RouteViews.
Here is a step-by-step guide for deploying BGPKIT Broker with Docker
- Checkout the repository
- run
docker-compose -f deployment/docker-compose.yml -p bgpkit-broker up
The initial database bootstrap phase would take about 30 minutes to one hour depending on your deployment environment. It requires at least 30GB of free storage for the container and build processes.
After the initial bootstrap phase is done, the API service should be up and
running, currently hosted at port 18888. You can modify the port in
docker-compose.yml file. In the mean time, a cronjob service also started,
crawling collectors for recent data every 5 minutes. The frequency can be
configured in the update.cron. It is not recommended to go more frequent than
one crawl per 5 minutes.
You can check out if the API is running by running:
curl 0.0.0.0:18888/latest
curl "0.0.0.0:18888/search?ts_start=1643673600&ts_end=2022-02-02&collector_id=rrc00"BGPKIT Broker backend code is under MIT LICENSE.
BGPKIT is a small-team focuses on building the best open-source tooling for BGP data processing and analysis in Rust. Learn more about what we are offering at https://github.com/bgpkit.
