Bitcoin Block Arrival Time Dataset
A dataset of block arrival times at different nodes.
CSV Format
The timestamps.csv file contains the block height, hash, the millisecond
timestamp it arrived (not the timestamp in the header), and the name of the
source. It does not contain a CSV header.
Example:
height hash timestamp source
759483,00000000000000000002b74f1fd927768b2bfd52d2a5229826303d480db5c76a,1666250167839,0xB10C monitoring1
759483,00000000000000000002b74f1fd927768b2bfd52d2a5229826303d480db5c76a,1666250167000,0xb10c memo
759482,00000000000000000006e1c51365637bbc9a977b6bc027c00613025c1625a00a,1666249466074,0xB10C monitoring1
Adding Data
To add your block arrival times, create a CSV file with the above format.
Make sure the timestamps are in millisecond precision, and you've added a name
for the source. You can use cat timestamps.csv my-timestamps.csv > new-timestamps.csv
to create a new, unsorted timestamp file. To sort it, you can use
LC_ALL=C sort --reverse --unique new-timestamps.csv > timestamps.csv.
Also, please remember to update the data-availability graph (see below).
Parsing Bitcoin Core debug.log
Block arrival timestamps can be parsed from the Bitcoin Core debug.log.
A Python tool is provided under contrib/process-debug-log.py. This expects
a debug.log as input file, a CSV file output name, and the name of the data
source.
python3 contrib/process-debug-log.py /home/b10c/.bitcoin/debug.log my-timestamps.csv "0xb10c node2"
Quality Assurance
The dataset is run through automatic quality assurance checks in the CI. A sanity check tests if the dataset is sorted and doesn't contain duplicate rows. Additionally, there is a check for the arrival timestamps. We assume these should be either two hours before or after the block header timestamp. For this, we maintain a list of height and header timestamps in "qa/block-timestamps/block-timestamps.csv". When adding new timestamps, the list might need to be updated. This can be done with the Bash script "qa/block-timestamps/update-block-timestamps.sh" requiring a Bitcoin Core instance with the REST server enabled.
A data availability graph can be generated with the tool
qa/data-availability/gen-mermaid.py. This should be updated when adding a
new data source.
gantt
dateFormat x
title data availability (not showing potential per-source holes)
todayMarker off
darosior node0: 1676970378000, 1702022714000
0xB10C monitoring1: 1605513978094, 1701968604513
0xb10c memo: 1635945250000, 1666268995000
0xb10c memo-old: 1562848974000, 1605772694000