=================
== mellowd.dev ==
=================
Darren's random posts

Migrating from bird to bird2 - part 2

RPKI-RTR

bird2 now supports the RPKI-RTR protocol. Instead of manually creating ROAs and refreshing bird, a cache server can now send ROAs directly to bird over TCP. This does mean I need to run a cache server, but long term this really is better. To understand why it may be useful to go over how I get ROAs working with bird.

First, create a ROA table in bird.conf and bird6.conf:

# IPv4 RPKI ROAs
roa table roa_table {
  include "/etc/bird/v4.roa";
}
# IPv6 RPKI ROAs
roa table roa_table {
  include "/etc/bird/v6.roa";
}

The above pulls in a list of ROAs that you can validate on. But I require to get those ROAs in the first place. To do this I created getroa. This would download the current list of ROAs from cloudflare, read in the json, and output the above v4.roa and v6.roa files. It would then tell bird to refresh to read in the new ROAs.

I then create a cronjob to run every six hours.

###############
# Update ROAs #
###############
# Every day, every six hours at 15 minutes past
15 3,9,15,22 * * * /home/bgp/getroa/getroa

To be fair, this is a perfectly fine solution. It doesn’t really matter if ROAs are a couple of hours out of date in my set up. It also uses minimal RAM as it’s simply updating two files then telling bird to refresh.

However I did want to move to RPKI-RTR anyway. This allows the cache to continuously update bird2 without having to refresh anything. One important point to realise is that RPKI-RTR is also not instant in that the RFC states the cache should only update every hour.

Routinator 3000

There are a few cache servers available right now, but I’ve settled on routinator 3000 by NLNETLABS for now. Routinator 3000 runs as a daemon, rsyncs the latest RPKI data from each RIR once an hour, and provides ROA updae to BGP listeners over TCP. Setting it up is quite easy, and I recommend you got read the docs to get it installed.

Connecting bird2

Now that you have a routinator cache set up, create your ROA tables in bird2 and connect it to your cache. In my case here I have two routinator instances set up and point bird to both.

##########
## ROAs ##
##########
roa4 table roa_v4;
roa6 table roa_v6;

protocol rpki routinator1 {
  roa4 { table roa_v4; };
  roa6 { table roa_v6; };
  remote "x.x.x.x" port 3323;
  retry keep 90;
  refresh keep 900;
  expire keep 172800;
}

protocol rpki routinator2 {
  roa4 { table roa_v4; };
  roa6 { table roa_v6; };
  remote "y.y.y.y" port 3323;
  retry keep 90;
  refresh keep 900;
  expire keep 172800;
}

bird2 will show this as just another protocol connected.

bird> show protocols all routinator2
Name       Proto      Table      State  Since         Info
routinator2 RPKI       ---        up     13:14:54.500  Established
  Cache server:     y.y.y.y:3323
  Status:           Established
  Transport:        Unprotected over TCP
  Protocol version: 1
  Session ID:       59247
  Serial number:    0
  Last update:      before 502.164 s
  Refresh timer   : 397.835/900
  Retry timer     : ---
  Expire timer    : 172297.835/172800
  Channel roa4
    State:          UP
    Table:          roa_v4
    Preference:     100
    Input filter:   ACCEPT
    Output filter:  REJECT
    Routes:         95818 imported, 0 exported, 95818 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:         191682          0          0      95818      95864
      Import withdraws:           46          0        ---          0         46
      Export updates:              0          0          0        ---          0
      Export withdraws:            0        ---        ---        ---          0
  Channel roa6
    State:          UP
    Table:          roa_v6
    Preference:     100
    Input filter:   ACCEPT
    Output filter:  REJECT
    Routes:         16221 imported, 0 exported, 16221 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:          32453          0          0      16221      16232
      Import withdraws:           11          0        ---          0         11
      Export updates:              0          0          0        ---          0
      Export withdraws:            0        ---        ---        ---          0

You can get both IPv4 and IPv6 ROAs over a single IPv4 or IPv6 connection. Here you can see I have 95818 IPv4 ROAs and 16221 IPv6 ROAs from the cache server.

You can view each roa table directly

bird> show route table roa_v4
Table roa_v4:
202.144.46.0/24-24 AS9583  [routinator2 2019-10-20] * (100)
58.69.253.0/24-24 AS36776  [routinator2 2019-10-20] * (100)
37.9.220.0/22-22 AS57626  [routinator2 2019-10-20] * (100)

Doing a show route count now also shows how many routes in each table

bird> show route count
7772754 of 7772754 routes for 798581 networks in table master4
700080 of 700080 routes for 76999 networks in table master6
95818 of 95818 routes for 95818 networks in table roa_v4
16221 of 16221 routes for 16221 networks in table roa_v6
Total: 8584873 of 8584873 routes for 987619 networks in 4 tables

Advantages/Disadvantages

Using RPKI-RTR means I no longer need to manually craft a list of ROAs and insert them into bird. However I now need to run another binary and all the resources associated with that binary. Routinator states it’s targetting low memory, however I’m seeing it use roughly 700MB+ right now. Considering 10 full BGP tables on both IPv4 and IPv6 is taking roughly 1.2GB, this 700MB seems exessive. I’m not sure this usage is correct and I have an open thread on it on the routinator mailing list.

I’m also running a routinator instance on all of my BGP collectors, each of which has limited resources. I’d really like to run a routinator instance separately that all my bird servers can attach to. It would be even better if someone was running a publically accessible instance that I could connect to ;)

That’s part 2 down. In part 3 I’ll go over the specific changes I’ve needed to make for bgpstuff.net, @bgp4_table, and @bgp6_table