After creating my first (okay, third ;)) VPC, in the previous post: Flow Virtual Networking – Basic VPC, I wanted to take a look at dynamic routing. Since BGP is the supported dynamic routing protocol for FVN, that is the way to go. I already configured my pfSense with FRR BGP, so there I only need to add the new neigbors, but on the Prism Central side, there is some more configuring to do.

There is also something to take into account, BGP on FVN doesn’t work as I was used to. With NSX, the BGP peering would be done by components that are also part of the datapath. So the next hop for a packet (both sides) would be advertised by itself. But with FVN there is a BGP controlplane and a BGP dataplane.

It feels a little bit like NSX-V used to do with OSPF. There was a VM (“ESG or Edge Services Gateway”) that would be responsible for dealing with OSPF connectivity and the datapath would be distributed. In a way, the method that FVN uses for BGP is like that.

The big benefit of this, is that the datapath is distributed. The “downside” is that configuration (at least for me) took some getting my head around it.

So we set up the following environment (building on the previous setup):

First, we create a new subnet, that will be used to host the BGP Control Plane. This subnet is not necessary per se, you could use an existing, VLAN-based subnet, but I wanted to be able to control all the parts in this exercise. So I created:

I also created an interface on the pfSense, in the same VLAN, with IP-Address: 172.30.9.254.

Next, I created the Gateway. This is done within Prism Central, under:

At this time, we create a Local Gateway.

We select the Subnet the Gateway will be connected to (the one we just created):

And then we specify we are creating a BGP Gateway Service (as opposed to a VPN or VTEP).

We bind it to two static IP Addresses (for the number of Hosts that I have configured earlier, if you have more hosts, you will need more IP Addresses ánd more BGP peers). Also I use a new ASN (65103) that does not conflict with my earlier configuration.

and then I click “Create”.

It’s interesting that I need to also include the prefix of the static IP Addresses. Since I already selected that when creating the subnet, I expected that not to be necessary, but it won’t let me move forward if I only put in the IP Addresses themselves.

Next, I create an external Gateway Service (BGP) (from the same screen where I created the local Gateway Service):

Note that I am using the IP address for the pfSense that is in the datapath, since the dataplane and controlplane are only separated on the FVN environment.

After creating this, I can create the BGP Sessions:

And do the same on the pfSense. On the pfSense it is important to set the Password as follows:

Otherwise the connection will stay down.

Another important thing to note, is that the IP Address that is used for the BGP connection, from within the pfSense is changed to the address that is part of the dataplane (as explained earlier):

If that is left to the “Default” setting, the routes that will be advertised to the FVN environment, will have a next hop of (in my case) 172.30.9.254 instead of 172.30.2.254. The consequence of this would be that the routes that are received by the gateway within FVN will be ignored, because the subnet that is used for the external connectivity is on the 172.30.2.0/24 network:

After changing the setting within the pfSense, the routes get added to the routing and forwarding table (note that the Next hop now is on the 172.30.2.0/24 network instead of the 172.30.9.0/24 network):

After all the configuration has been done as it should (don’t forget to set Prefix List Filter and Route Map Filters within pfSense), the connection is up and the routes get learned both on the site of FVN:

and on the pfSense:

When I then add a new subnet, with an IP range that is outside of the scope of the VPC:

it is not automatically added to the routing table of the pfSense. It is not advertised by FVN. When I want to have that subnet advertised as well, I need to add it to the VPC03’s list of Externally Routable IP Addresses:

Once that is done, it shows up on the pfSense as well:

And when I create a VM in that subnet, I can reach it from my laptop:

and from within the VM, I can ping the world:

At the version that I am running (due to the fact that I am on the Community Edition of AOS), I have not a lot of options to play around with BGP, so this is basically it. We can set up BGP routing and have some basic BGP functionality.

That being said, in the newer versions of NCI (7.3), a lot of new functionality is added, to steer BGP.

---

One last thing I did was also put a VM on the second cluster (Ronald-Cluster2) on the newly created network and power it on:

After it booted, I can reach it from my PC and I can also reach it from the other VM that is running on the other cluster. That was pretty straightforward, to be honest…