Migrating from bird to bird2 - part 2
RPKI-RTR
bird2 now supports the RPKI-RTR protocol. Instead of manually creating ROAs and refreshing bird, a cache server can now send ROAs directly to bird over TCP. This does mean I need to run a cache server, but long term this really is better. To understand why it may be useful to go over how I get ROAs working with bird.
First, create a ROA table in bird.conf and bird6.conf:
# IPv4 RPKI ROAs
roa table roa_table {
include "/etc/bird/v4.roa";
}
# IPv6 RPKI ROAs
roa table roa_table {
include "/etc/bird/v6.roa";
}
The above pulls in a list of ROAs that you can validate on. But I require to get those ROAs in the first place. To do this I created getroa. This would download the current list of ROAs from cloudflare, read in the json, and output the above v4.roa and v6.roa files. It would then tell bird to refresh to read in the new ROAs.
I then create a cronjob to run every six hours.
###############
# Update ROAs #
###############
# Every day, every six hours at 15 minutes past
15 3,9,15,22 * * * /home/bgp/getroa/getroa
To be fair, this is a perfectly fine solution. It doesn’t really matter if ROAs are a couple of hours out of date in my set up. It also uses minimal RAM as it’s simply updating two files then telling bird to refresh.
However I did want to move to RPKI-RTR anyway. This allows the cache to continuously update bird2 without having to refresh anything. One important point to realise is that RPKI-RTR is also not instant in that the RFC states the cache should only update every hour.
Routinator 3000
There are a few cache servers available right now, but I’ve settled on routinator 3000 by NLNETLABS for now. Routinator 3000 runs as a daemon, rsyncs the latest RPKI data from each RIR once an hour, and provides ROA updae to BGP listeners over TCP. Setting it up is quite easy, and I recommend you got read the docs to get it installed.
Connecting bird2
Now that you have a routinator cache set up, create your ROA tables in bird2 and connect it to your cache. In my case here I have two routinator instances set up and point bird to both.
##########
## ROAs ##
##########
roa4 table roa_v4;
roa6 table roa_v6;
protocol rpki routinator1 {
roa4 { table roa_v4; };
roa6 { table roa_v6; };
remote "x.x.x.x" port 3323;
retry keep 90;
refresh keep 900;
expire keep 172800;
}
protocol rpki routinator2 {
roa4 { table roa_v4; };
roa6 { table roa_v6; };
remote "y.y.y.y" port 3323;
retry keep 90;
refresh keep 900;
expire keep 172800;
}
bird2 will show this as just another protocol connected.
bird> show protocols all routinator2
Name Proto Table State Since Info
routinator2 RPKI --- up 13:14:54.500 Established
Cache server: y.y.y.y:3323
Status: Established
Transport: Unprotected over TCP
Protocol version: 1
Session ID: 59247
Serial number: 0
Last update: before 502.164 s
Refresh timer : 397.835/900
Retry timer : ---
Expire timer : 172297.835/172800
Channel roa4
State: UP
Table: roa_v4
Preference: 100
Input filter: ACCEPT
Output filter: REJECT
Routes: 95818 imported, 0 exported, 95818 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 191682 0 0 95818 95864
Import withdraws: 46 0 --- 0 46
Export updates: 0 0 0 --- 0
Export withdraws: 0 --- --- --- 0
Channel roa6
State: UP
Table: roa_v6
Preference: 100
Input filter: ACCEPT
Output filter: REJECT
Routes: 16221 imported, 0 exported, 16221 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 32453 0 0 16221 16232
Import withdraws: 11 0 --- 0 11
Export updates: 0 0 0 --- 0
Export withdraws: 0 --- --- --- 0
You can get both IPv4 and IPv6 ROAs over a single IPv4 or IPv6 connection. Here you can see I have 95818 IPv4 ROAs and 16221 IPv6 ROAs from the cache server.
You can view each roa table directly
bird> show route table roa_v4
Table roa_v4:
202.144.46.0/24-24 AS9583 [routinator2 2019-10-20] * (100)
58.69.253.0/24-24 AS36776 [routinator2 2019-10-20] * (100)
37.9.220.0/22-22 AS57626 [routinator2 2019-10-20] * (100)
Doing a show route count now also shows how many routes in each table
bird> show route count
7772754 of 7772754 routes for 798581 networks in table master4
700080 of 700080 routes for 76999 networks in table master6
95818 of 95818 routes for 95818 networks in table roa_v4
16221 of 16221 routes for 16221 networks in table roa_v6
Total: 8584873 of 8584873 routes for 987619 networks in 4 tables
Advantages/Disadvantages
Using RPKI-RTR means I no longer need to manually craft a list of ROAs and insert them into bird. However I now need to run another binary and all the resources associated with that binary. Routinator states it’s targetting low memory, however I’m seeing it use roughly 700MB+ right now. Considering 10 full BGP tables on both IPv4 and IPv6 is taking roughly 1.2GB, this 700MB seems exessive. I’m not sure this usage is correct and I have an open thread on it on the routinator mailing list.
I’m also running a routinator instance on all of my BGP collectors, each of which has limited resources. I’d really like to run a routinator instance separately that all my bird servers can attach to. It would be even better if someone was running a publically accessible instance that I could connect to ;)
That’s part 2 down. In part 3 I’ll go over the specific changes I’ve needed to make for bgpstuff.net, @bgp4_table, and @bgp6_table