I learned something today, which in hindsight is obvious. Hopefully this helps someone that runs into the same “strange” (but not so strange) behavior.

I created the following topology today, to prepare for some NSX-T demo I am giving tomorrow:

What I (among other stuff) wanted to show, was that routing between Test-Segments “D” and “E” and “A”, “B” and “C”, is completely distributed. So when VM’s from the different segments live on the same host, no physical hops are in the path.

To my surprise, after I created the environment, there wás hopping going on. Even when the VM’s where on the same host, the traffic would go from the host to the edge-node on which the T1 was hosted and from there back to the host.

So below is a picture of a TraceFlow which is using the T1’s SR to route traffic:

As you can see (sorry for the small print in the picture), the traffic is going out of the “esx” host and towards the “edge” (hop 1) and later on, it goes back to the “esx” host again (hop 2). You have to trust me that the host is the same on both occasions.

So after asking around, one of my instructors during the LiveFire training last month, suggested that the T1’s might be located on an Edge Node and that that would instantiate the T1’s SR. Since there could be firewalling or other stateful services used in the traffic, it would make sense that traffic is put through the SR and not just through the DR’s.

When I tried to delete the Edge Node Cluster config from the T1, I got the following error:

which basically tells me that, although I didn’t configure it, the firewall service is active on this T1, when it is connected to an Edge Node Cluster. When looking at the advanced configuration of the T1, I can verify this:

So in this case, I deleted the T1’s (after disconnecting the Segments off course) and recreated them, this time not on an Edge Node Cluster, and voila, routing is fully distributed:

And this also helps with reducing latency :).

So, it works as should be expected, but it took some time to figure that out and why. Thanks to my trainer at the NSX-T LiveFire (which I can highly recommended!).