Unintuitive ts_start behaviour

[Cinderella-Man]

Hello 👋

I was trying to run the following URL:

curl -X 'GET'
'https://alpha.api.bgpkit.com/broker?ts_start=1684915200&project=riperis&collectors=rrc00&data_type=updates&page_size=1'
-H 'accept: application/json'

using the Swagger UI: https://alpha.api.bgpkit.com/docs/#/meta/search_peer_stats

But my very first result was:

{
  "page": 0,
  "page_size": 1,
  "count": 1,
  "data": [
    {
      "ts_start": "2023-05-24T07:55:00",
      "ts_end": "2023-05-24T08:00:00",
      "project": "riperis",
      "collector": "rrc00",
      "data_type": "update",
      "url": "https://data.ris.ripe.net/rrc00/2023.05/updates.20230524.0755.gz",
      "size": 5242880
    }
  ]
}

The ts_start timestamp that I'm using 1684915200 is "2023-05-24T08:00:00" - I'm not sure why I am getting a record with "ts_start": "2023-05-24T07:55:00",? Is this expected?

Thanks in advance for looking into this

[digizeph]

Hey @Cinderella-Man Thank for submitting this issue!

The timestamp alignment behavior you mentioned is the current expected behavior. There has been some discussion about this matter previously (see thread here). The main reason for the current behavior is to avoid potentially missing BGP messages that happens on the exact timestamp and saved in the previous MRT file. In other words, the previous file is included because the ts_end lands on the exact start time of the query. I am more convinced now that this is actually not happening in real life, and plan to change the behavior in the upcoming release. In the mean time, you can add a second to your query and it should return the update files you expected.

By the way, we have also a post-alpha API endpoint available now at https://api.bgpkit.com/docs. The main difference is that the API is now deployed on Cloudflare (instead of a bare-metal Python instance), which has better performance and more capable on handling concurrent queries. Your query in the above example will be

https://api.bgpkit.com/broker/search?ts_start=1684915200&project=riperis&collectors=rrc00&data_type=updates&page_size=1

in the new endpoint (all query parameters remain the same).

You are welcome to join our Discord server and ping me there if you would like to DM me.

[Cinderella-Man]

Hey @digizeph

I appreciate you taking the time to answer and explain the reasoning behind the decision - it makes sense now, although, as you pointed out, the mentioned edge case is not really happening - I guess dropping it will save you some time re-explaining it in the future 👍

For the time being, I will just add one to the timestamp - thanks for the suggestion.

I will also start using the non-alpha API, as the one that I was trying has been lagging really badly from time to time.

Thank you once again! 🙏