/bgptest

Primary LanguagePython

BGPTEST

This is a set of tools and a simulation to stress test a BGP implmentation. 2 tools are used to test genrt.py and bgp-inject.py. Also included is a simulation setup that runs the test using docker to simulate 4 BGP routers.

Running Requirements

Requirements for using tools:

  • Ubuntu 18 with some packages installed.
    • sudo apt-get update -qy
    • sudo apt-get install bash python3 python3-dev python3-venv

Requirements for simulation

  • Additionally add latest docker package.
    • curl -fsSL get.docker.com | sh
    • sudo usermod -aG docker $(whoami)
    • logout and back in so you are part of the unix group.

Description of Tools

genrt.py

Generates routes in various formats. It can create raw BGP Update messages to inject directly into a BGP connection (using bgp-inject.py). It can also create a list of static route definitions that can be included in BIRD (or other) routing configs. Finally it can create MRT table dumps of routes that can be used with gobgp to inject routes. MRT table dumps are also used by RIPE and others to record BGP sessions and routing tables.

bgp-inject.py

bgp-inject.py Opens an EBGP connection and then sends the contents of file over the connection. The file content (BGP update messages) can be created by the genrt.py tool. Finally it enters a Keep Alive loop to keep the BGP connection established.

Running the simulation.

The physical test is intended to be run with 4

+---------------------------------+        +------+
|              bgput              |--------| ibgp |
+---------------------------------+  IBGP  +------+
    |        |        |       |
    |        |  EBGP  |       |
    |        |        |       |
+------+ +------+ +------+ +------+
| st-1 | | st-1 | | st-1 | |  ft  |
+------+ +------+ +------+ +------+

To generate all the required configs and bringup the simulation one can simply

make run

All the configs and update messages files and 4 routers are created:

  • bgput - The BGP under test everything connects to this router (AS30).
  • st-1 - BGP Static 1 - 2.5M routes (AS10)
  • st-2 - BGP Static 2 - 2.5M routes (AS11)
  • st-3 - BGP Static 3 - 2.5M routes (AS12)
  • ibgp - IBGP speaker to check for presence of routes from bgput

One can use the birdc6 CLI tool to examine state of the routers. For example to look at the protocol stats (including route counts) you could use the following command

docker-compose exec ibgp birdc6 show proto all

Or to just look at the BGP speaker on IBGP

docker-compose exec ibgp birdc6 show proto all bgp_30

The fifth router (FT) is you, running the bgp-inject.py tool directly from the shell. This would be similar to testing a real router setup.

To bring down the simualtion use make stop.

Checking Results On The Simulation.

To check the wall clock time that it takes for the simulation to launch and stabilize (or for the injected routes to be processed) one can run the following command. This can be run prior to launching the simulation.

make watch

The command is watching for the number of routes received by the IBGP neighbor from the BGP under test. (bgput). On launch all of these routes will be from the static configuration EBGP neighbors of bgput, namely, st-1, st-1, st-1, who each contribute 2.5M routes.

For a physical router setup a similar command to what make watch is running should be used to query the installed routes on the ibgp router. See the Makefile for what the watch target is doing.

Running the test.

These instructions assume the simulation has been launched with make run from above. In a physical router setup the connectivity would need to be in place and the addresses adjusted accordingly (if needed).

After the update messages raw file has been generated and the routers (or simulation) brought up you are ready to inject the routes over EBGP this can be done with the following command

./bgp-inject.py -i data/mpath-0-1000.raw fc20::1,30

This creates an EBGP speaker AS 20 router ID 10.0.0.20, connecting to fc20::1 remote AS 30. Once connection is ESTABLISHED the tool will send the content of the file (data/mpath-0-1000.raw in this case) to the remote speaker.

We generate a few variants of 2.5M route updates. Here’s the format:

<type>-<mod>-<pack>

The type is either

  • onepath - uses the same set of AS path for all routes
  • mpath - the first AS in the path is incremented for each update message.

The mod is a modulus applied to the incremented AS number. This effectively limits the number of unique AS paths to this value. 0 means no modulus is applied.

  • 0 mod no modulus is done all paths will be unique (if incrementing)
  • 100 mod this means that every after 100 increments the AS number will wrap so there will be 100 unique AS paths.
  • 1000 mod this means that every after 1000 increments the AS number will wrap so there will be 1000 unique AS paths.

The mpack value is the number of prefixes to pack into a single update message. By definition all these prefix will share the same path.

We generated the following files automatically:

  • data/mpath-0-1.raw
  • data/mpath-0-10.raw
  • data/mpath-0-1000.raw
  • data/mpath-100-1.raw
  • data/mpath-100-10.raw
  • data/mpath-100-1000.raw
  • data/mpath-1000-1.raw
  • data/mpath-1000-10.raw
  • data/mpath-1000-1000.raw
  • data/onepath-0-1.raw
  • data/onepath-0-10.raw
  • data/onepath-0-1000.raw

Run Notes:

Shell cmd to query IBGP:

Use this command to query the IBGP instance in a loop looking at route counts. This is used below to track route count during the runs.

while sleep 1; do \
  printf "%s: " "$(date)"; \
  docker-compose exec ibgp birdc6 show proto all bgp_30 | grep Routes:; \
  done

2.5M routes, 1000 paths 100 prefix per update.

Letting the simulation come up and settle, then running the inject with 1000 unique paths and 100 prefix per update (thus using same path) takes

  • ~14s to send the raw data:
  • ~18s to see them in BIRD iBGP instance.

The sending time is definitely affected by the redistribution in BIRD. The next section after this one shows a test with just the BGPUT and the FT injecting routes.

Output when using ./bgp-inject.py -i data/mpath-1000-100.raw fc20::1,30

[ simulation startup ]
Sat Jul 28 11:32:08 EDT 2018:   Routes:         0 imported, 0 exported, 0 preferred
Sat Jul 28 11:32:09 EDT 2018:   Routes:         6 imported, 1 exported, 4 preferred
Sat Jul 28 11:32:11 EDT 2018:   Routes:         788361 imported, 1 exported, 788359 preferred
Sat Jul 28 11:32:13 EDT 2018:   Routes:         1992842 imported, 1 exported, 1992840 preferred
Sat Jul 28 11:32:15 EDT 2018:   Routes:         3092105 imported, 1 exported, 3092103 preferred
Sat Jul 28 11:32:17 EDT 2018:   Routes:         4286332 imported, 1 exported, 4286330 preferred
Sat Jul 28 11:32:18 EDT 2018:   Routes:         5515840 imported, 1 exported, 5515838 preferred
Sat Jul 28 11:32:20 EDT 2018:   Routes:         6746259 imported, 1 exported, 6746257 preferred
Sat Jul 28 11:32:21 EDT 2018:   Routes:         7864326 imported, 1 exported, 7864324 preferred
Sat Jul 28 11:32:23 EDT 2018:   Routes:         7864326 imported, 1 exported, 7864324 preferred
Sat Jul 28 11:32:24 EDT 2018:   Routes:         7864326 imported, 1 exported, 7864324 preferred
Sat Jul 28 11:32:26 EDT 2018:   Routes:         7864326 imported, 1 exported, 7864324 preferred
Sat Jul 28 11:32:27 EDT 2018:   Routes:         7935026 imported, 1 exported, 7935024 preferred
[ Started ./bgp-inject.py ]
Sat Jul 28 11:32:29 EDT 2018:   Routes:         8358526 imported, 1 exported, 8358524 preferred
Sat Jul 28 11:32:30 EDT 2018:   Routes:         8650214 imported, 1 exported, 8650212 preferred
Sat Jul 28 11:32:32 EDT 2018:   Routes:         8910014 imported, 1 exported, 8910012 preferred
Sat Jul 28 11:32:33 EDT 2018:   Routes:         9133502 imported, 1 exported, 9133500 preferred
Sat Jul 28 11:32:35 EDT 2018:   Routes:         9335502 imported, 1 exported, 9335500 preferred
Sat Jul 28 11:32:36 EDT 2018:   Routes:         9534002 imported, 1 exported, 9534000 preferred
Sat Jul 28 11:32:38 EDT 2018:   Routes:         9726502 imported, 1 exported, 9726500 preferred
Sat Jul 28 11:32:39 EDT 2018:   Routes:         9921002 imported, 1 exported, 9921000 preferred
Sat Jul 28 11:32:41 EDT 2018:   Routes:         10129078 imported, 1 exported, 10129076 preferred
Sat Jul 28 11:32:42 EDT 2018:   Routes:         10363078 imported, 1 exported, 10363076 preferred
Sat Jul 28 11:32:44 EDT 2018:   Routes:         10485766 imported, 1 exported, 10485764 preferred
Sat Jul 28 11:32:45 EDT 2018:   Routes:         10485766 imported, 1 exported, 10485764 preferred
Sat Jul 28 11:32:47 EDT 2018:   Routes:         10485766 imported, 1 exported, 10485764 preferred

Output from ./bgp-inject.py -i data/mpath-1000-100.raw fc20::1,30

[...]
2018-07-28 11:32:27,860: INFO: ESTABLISHED (IPv6Address('fc20::1'), 179) (assuming 4-octet AS numbers)
2018-07-28 11:32:27,860: INFO: Loading raw data from data/mpath-1000-100.raw
2018-07-28 11:32:27,877: INFO: Sending 19529767 bytes of raw data to (IPv6Address('fc20::1'), 179)
2018-07-28 11:32:42,215: INFO: Done sending after 14.355535507202148 to (IPv6Address('fc20::1'), 179)

FT+BGPUT only: 2.5M routes, 1000 paths 100 prefix per update.

You can see here that when there is no other neighbors and thus no redistribution, it takes about 3 seconds to send the 2.5M prefix this is above 500k (app 853k updates/s)

Sat Jul 28 12:09:29 EDT 2018:   Routes:         0 imported, 0 exported, 0 preferred
Sat Jul 28 12:09:30 EDT 2018:   Routes:         923288 imported, 0 exported, 923288 preferred
Sat Jul 28 12:09:32 EDT 2018:   Routes:         2171152 imported, 0 exported, 2171152 preferred
Sat Jul 28 12:09:34 EDT 2018:   Routes:         2621440 imported, 0 exported, 2621440 preferred

(default) 06:00 [12:08:23 tops:~/w/bgptest]$ ./bgp-inject.py -i data/mpath-1000-100.raw fc20::1,30
2018-07-28 12:09:31,033: INFO: CONNECT (IPv6Address('fc20::1'), 179)
2018-07-28 12:09:31,034: INFO: SENT OPEN (IPv6Address('fc20::1'), 179)
2018-07-28 12:09:31,034: INFO: OPENCONFRIM (IPv6Address('fc20::1'), 179)
2018-07-28 12:09:31,034: INFO: ESTABLISHED (IPv6Address('fc20::1'), 179) (assuming 4-octet AS numbers)
2018-07-28 12:09:31,034: INFO: Loading raw data from data/mpath-1000-100.raw
2018-07-28 12:09:31,052: INFO: Sending 19529767 bytes of raw data to (IPv6Address('fc20::1'), 179)
2018-07-28 12:09:34,119: INFO: Done sending after 3.085615873336792 to (IPv6Address('fc20::1'), 179)
2018-07-28 12:09:34,120: INFO: Sent KeepAlive to (IPv6Address('fc20::1'), 179) sleeping 30s

FT+BGPUT only: 2.5M routes, all unique paths 1 prefix per update.

If we send all unique paths and only a single prefix per update message (the most stressful and a somewhat unrealistic test) the time jumps to ~8s slightly (~300k updates/s)

Sat Jul 28 12:13:56 EDT 2018:   Routes:         0 imported, 0 exported, 0 preferred
Sat Jul 28 12:13:58 EDT 2018:   Routes:         416730 imported, 0 exported, 416730 preferred
Sat Jul 28 12:14:00 EDT 2018:   Routes:         1271654 imported, 0 exported, 1271654 preferred
Sat Jul 28 12:14:01 EDT 2018:   Routes:         1829414 imported, 0 exported, 1829414 preferred
Sat Jul 28 12:14:03 EDT 2018:   Routes:         2196810 imported, 0 exported, 2196810 preferred
Sat Jul 28 12:14:05 EDT 2018:   Routes:         2561020 imported, 0 exported, 2561020 preferred
Sat Jul 28 12:14:07 EDT 2018:   Routes:         2621440 imported, 0 exported, 2621440 preferred

(default) 00:01 [12:13:09 tops:~/w/bgptest]$ ./bgp-inject.py -i data/mpath-0-1.raw fc20::1,30
2018-07-28 12:13:58,370: INFO: CONNECT (IPv6Address('fc20::1'), 179)
2018-07-28 12:13:58,370: INFO: SENT OPEN (IPv6Address('fc20::1'), 179)
2018-07-28 12:13:58,370: INFO: OPENCONFRIM (IPv6Address('fc20::1'), 179)
2018-07-28 12:13:58,370: INFO: ESTABLISHED (IPv6Address('fc20::1'), 179) (assuming 4-octet AS numbers)
2018-07-28 12:13:58,370: INFO: Loading raw data from data/mpath-0-1.raw
2018-07-28 12:13:58,526: INFO: Sending 188219392 bytes of raw data to (IPv6Address('fc20::1'), 179)
2018-07-28 12:14:06,354: INFO: Done sending after 7.983449935913086 to (IPv6Address('fc20::1'), 179)
2018-07-28 12:14:06,354: INFO: Sent KeepAlive to (IPv6Address('fc20::1'), 179) sleeping 30s

Not Used, Too Slow Notes

Other BGP implementations were tested before settling on BIRD6 as the BGP of choice for the simulation. These other BGPs were significantly slower than BIRD which is why they were not chosen.

FRR - BGPD

Testing with FRR BGPD as bgput and bird6 as all other routers (st-{1,2,3}, ibgp).

In order to test with the latest FRR one should rebuild the docker test image locally with:

make build

In order to run the test with FRR one should pass FRR=1 to the make command.

make FRR=1 run

NOTE The injection test fails with BGPD. The connection is opened and the updates are sent but not processed. This should be investigated further; however, the launch test should serve as a test for now to improve timings.

Launch timing.

FRR launch/stabilize time is ~4 times slower than bird (app. 61s vs 17s)

(default) [05:54:38 dak:~/w/bgptest]$ make watch
while sleep 1; do \
        printf "%s: " "$(date)"; \
        docker-compose exec ibgp birdc6 show proto all bgp_30 | grep Routes:; \
done
Sat Jul  4 05:56:00 EDT 2020: ERROR: No container found for ibgp_1
Sat Jul  4 05:56:01 EDT 2020: ERROR: No container found for ibgp_1
Sat Jul  4 05:56:03 EDT 2020:   Routes:         0 imported, 0 exported, 0 preferred
Sat Jul  4 05:56:04 EDT 2020:   Routes:         0 imported, 1 exported, 0 preferred
Sat Jul  4 05:56:06 EDT 2020:   Routes:         0 imported, 1 exported, 0 preferred
Sat Jul  4 05:56:08 EDT 2020:   Routes:         0 imported, 1 exported, 0 preferred
Sat Jul  4 05:56:09 EDT 2020:   Routes:         0 imported, 1 exported, 0 preferred
Sat Jul  4 05:56:11 EDT 2020:   Routes:         202496 imported, 1 exported, 202496 preferred
Sat Jul  4 05:56:13 EDT 2020:   Routes:         448256 imported, 1 exported, 448256 preferred
Sat Jul  4 05:56:15 EDT 2020:   Routes:         686336 imported, 1 exported, 686336 preferred
Sat Jul  4 05:56:16 EDT 2020:   Routes:         939776 imported, 1 exported, 939776 preferred
Sat Jul  4 05:56:18 EDT 2020:   Routes:         1170176 imported, 1 exported, 1170176 preferred
Sat Jul  4 05:56:20 EDT 2020:   Routes:         1446656 imported, 1 exported, 1446656 preferred
Sat Jul  4 05:56:21 EDT 2020:   Routes:         1700096 imported, 1 exported, 1700096 preferred
Sat Jul  4 05:56:23 EDT 2020:   Routes:         1953536 imported, 1 exported, 1953536 preferred
Sat Jul  4 05:56:25 EDT 2020:   Routes:         2130176 imported, 1 exported, 2130176 preferred
Sat Jul  4 05:56:26 EDT 2020:   Routes:         2398976 imported, 1 exported, 2398976 preferred
Sat Jul  4 05:56:28 EDT 2020:   Routes:         2647296 imported, 1 exported, 2647296 preferred
Sat Jul  4 05:56:30 EDT 2020:   Routes:         2890496 imported, 1 exported, 2890496 preferred
Sat Jul  4 05:56:31 EDT 2020:   Routes:         3143936 imported, 1 exported, 3143936 preferred
Sat Jul  4 05:56:33 EDT 2020:   Routes:         3389696 imported, 1 exported, 3389696 preferred
Sat Jul  4 05:56:35 EDT 2020:   Routes:         3635456 imported, 1 exported, 3635456 preferred
Sat Jul  4 05:56:36 EDT 2020:   Routes:         3869023 imported, 1 exported, 3869023 preferred
Sat Jul  4 05:56:38 EDT 2020:   Routes:         4103936 imported, 1 exported, 4103936 preferred
Sat Jul  4 05:56:40 EDT 2020:   Routes:         4226816 imported, 1 exported, 4226816 preferred
Sat Jul  4 05:56:41 EDT 2020:   Routes:         4472576 imported, 1 exported, 4472576 preferred
Sat Jul  4 05:56:43 EDT 2020:   Routes:         4697856 imported, 1 exported, 4697856 preferred
Sat Jul  4 05:56:45 EDT 2020:   Routes:         4948736 imported, 1 exported, 4948736 preferred
Sat Jul  4 05:56:46 EDT 2020:   Routes:         5202176 imported, 1 exported, 5202176 preferred
Sat Jul  4 05:56:48 EDT 2020:   Routes:         5447936 imported, 1 exported, 5447936 preferred
Sat Jul  4 05:56:50 EDT 2020:   Routes:         5693696 imported, 1 exported, 5693696 preferred
Sat Jul  4 05:56:51 EDT 2020:   Routes:         5939456 imported, 1 exported, 5939456 preferred
Sat Jul  4 05:56:53 EDT 2020:   Routes:         6185216 imported, 1 exported, 6185216 preferred
Sat Jul  4 05:56:55 EDT 2020:   Routes:         6415824 imported, 1 exported, 6415824 preferred
Sat Jul  4 05:56:56 EDT 2020:   Routes:         6665134 imported, 1 exported, 6665134 preferred
Sat Jul  4 05:56:58 EDT 2020:   Routes:         6922496 imported, 1 exported, 6922496 preferred
Sat Jul  4 05:56:59 EDT 2020:   Routes:         7168256 imported, 1 exported, 7168256 preferred
Sat Jul  4 05:57:01 EDT 2020:   Routes:         7390976 imported, 1 exported, 7390976 preferred
Sat Jul  4 05:57:03 EDT 2020:   Routes:         7631616 imported, 1 exported, 7631616 preferred
Sat Jul  4 05:57:04 EDT 2020:   Routes:         7864320 imported, 1 exported, 7864320 preferred
Sat Jul  4 05:57:06 EDT 2020:   Routes:         7864320 imported, 1 exported, 7864320 preferred

Compare this with a bird6 only run.

(default) [05:59:30 dak:~/w/bgptest]$ make watch
while sleep 1; do \
        printf "%s: " "$(date)"; \
        docker-compose exec ibgp birdc6 show proto all bgp_30 | grep Routes:; \
done
Sat Jul  4 06:01:47 EDT 2020: ERROR: No container found for ibgp_1
Sat Jul  4 06:01:49 EDT 2020: ERROR: No container found for ibgp_1
Sat Jul  4 06:01:50 EDT 2020:   Routes:         0 imported, 0 exported, 0 preferred
Sat Jul  4 06:01:52 EDT 2020:   Routes:         0 imported, 0 exported, 0 preferred
Sat Jul  4 06:01:54 EDT 2020:   Routes:         0 imported, 0 exported, 0 preferred
Sat Jul  4 06:01:55 EDT 2020:   Routes:         6 imported, 1 exported, 4 preferred
Sat Jul  4 06:01:57 EDT 2020:   Routes:         6 imported, 1 exported, 4 preferred
Sat Jul  4 06:01:59 EDT 2020:   Routes:         958454 imported, 1 exported, 958452 preferred
Sat Jul  4 06:02:01 EDT 2020:   Routes:         2302621 imported, 1 exported, 2302619 preferred
Sat Jul  4 06:02:02 EDT 2020:   Routes:         3773799 imported, 1 exported, 3773797 preferred
Sat Jul  4 06:02:04 EDT 2020:   Routes:         5254974 imported, 1 exported, 5254972 preferred
Sat Jul  4 06:02:06 EDT 2020:   Routes:         6688515 imported, 1 exported, 6688513 preferred
Sat Jul  4 06:02:07 EDT 2020:   Routes:         7864326 imported, 1 exported, 7864324 preferred
Sat Jul  4 06:02:09 EDT 2020:   Routes:         7864326 imported, 1 exported, 7864324 preferred

Go BGP

Some archival commands used when initially testing with gobgp.

Using GoBGP

# list neighbor
docker-compose exec bgput gobgp neighbor

# neighbor info
docker-compose exec bgput gobgp neighbor 10.0.0.20
docker-compose exec bgput gobgp neighbor 10.0.1.31

# Global RIB
gobgp global rib

# RIB-IN
gobgp neighbor 10.0.0.20 adj-in
gobgp neighbor 10.0.1.31 adj-in

# RIB-OUT
gobgp neighbor 10.0.0.20 adj-out
gobgp neighbor 10.0.1.31 adj-out