Creating stretched networks with NSX-T Federation

August 8, 2020 ronaldpj Comments 2 comments

After the initial configuration and setup of NSX-T local and global managers ánd setting up layer-3 communication (see previous blogs), we can go for the real new thing of NSX-T 3.0, stretched networking with Federation. We are aiming for the following setup:

In order to set this up, we need Edge Nodes on both sites (we have those) and a configuration of Remote Tunnel Endpoints (RTEPs). We also need to create a stretched T0 and T1 and setup BGP to the physical world. A lot to do.

Before we can do all that, we need to setup some prerequisites:

IP Address Pool for RTEPs (see the the initial configuration blog for that)
VLAN for RTEPs
VLAN’s for transit networks (connected to existing transport zones)

When we setup RTEPs, we need to use the Global NSX-T Manager. There we go to “System | Location Manager” and click “Networking” for the first Location:

And click “Configure”:

This will take us to the relevant local NSX Manager, which we can see here:

Here we fill in the relevant information for this location:

and click Save. Now we have setup the RTEPs for Site A, we can see (and change) the configuration here: “System | Fabric | Nodes | Edge Transport Nodes | Tunnels”:

To prepare for the stretched T0, we need to create the transit segments (VLAN based) to connect to the physical network. This can be done on the Global Manager, where we can select the location on which the segments will be created:

And this, four times, will lead to:

Now we need a Stretched Tier0-Gateway to connect the dots. This T0 needs to have interfaces on both edge node clusters (for both sites). Important to note is that we can’t use the same edge nodes for multiple T0’s. Only one T0 interface can be hosted on an edge node at the same time. Because I didn’t want to deploy additional edge nodes, I deleted the current T0, to replace it with this stretched T0, so the edge nodes will not be used for another T0 but the stretched one.

First, we create the T0. We select both sites and the edge node cluster we are using. We select both as Primary, this will make sure that the correct routes are used and everything is reachable.

And then we can configure the interfaces on the T0, bound to the specific edge nodes:

Again, four times, will lead to:

Now we can setup BGP routing from the new T0 to the physical network. For the stretched site, we are using AS 65040:

And make sure to correctly configure the route re-distribution, for both sites:

Next step, configure a stretched T1 and connect it to the T0:

And make sure we configure the route advertisement:

After we have done this, we can create a stretched segment:

This segment is visible on both VDS’s:

We can see that the edge nodes are connected through the proper tunnels:

And when we connect virtual machines from both sites to this segment, we can communicate between the two:

And to the outside world, through the T1 and T0:

Next up, we create two local T1’s, connected to the stretched T0 and segments connected to the T1’s, to complete the drawing at the beginning of the post (again, remember the route advertisements).

Then we get:

And:

When we look at the routing table on the physical router, we see the stretched network being advertised on all edge nodes, while the networks connected to the local T1 (which is connected to the stretched T0), we see that they are only available on the specific site:

(the fact that they are only active on one of the transit networks, is because my physical router doesn’t do ECMP (or I haven’t figured out how, yet ;)).

When we try to connect to a stretched virtual machine, running on Site A or B, we see one of the routes going over the “other site”, based on the fact that the second and third hop are identical: