Connecting external L2 segment to Cisco ACI using EPG and L2Out

Today I would like to share my experience in configuring Cisco ACI connectivity with an external L2 segment. As you know, there are two approaches to solving this problem: to classify the external segment into a separate EPG, or to use the External Bridged Network object, also known as L2Out.

The stand looks like this:

SW1 and SW2 are Nexus 3000 switches that act as fabric-attached hosts, so they only require L3 interfaces to test connectivity. APIC uses firmware version 4.2 (7l); specific models of factory switches do not matter much for the agenda. Access policies are also outside the scope of this article, so you can omit their configuration.

Let’s create the objects needed to ensure EPG connectivity: VRF and bridge domain (BD).

Figure 2. All VRF parameters - default — Figure 2. All VRF parameters – default

Since L3 connectivity is not required in this case, you can disable routing and uRPF within the BD, leaving the rest of the parameters unchanged:

Now we can create an application profile (AP) and start configuring L2 connectivity using the first approach: assign specific EPG ports SW1 and SW2, and then create the necessary contract between them.

Figure 6. Creating EPG, everything by default

It is important to set up the link between the EPG and the physical domain; otherwise, the policy will not be implemented in the factory. Let’s assign the physical ports of the switches to the corresponding EPG:

Figure 7. Purpose of the physical EPG port

Having (almost) finished the settings on the ACI side, let’s turn our attention to N3k:

SW1(config)# interface ethernet1/46
SW1(config-if)# shutdown
SW1(config-if)# no switchport
SW1(config-if)# ip add 192.168.0.1/24
SW1(config-if)# mac-address 0000.0000.0001

SW2(config)# interface ethernet1/46
SW2(config-if)# shutdown
SW2(config-if)# no switchport
SW2(config-if)# ip add 192.168.0.2/24
SW2(config-if)# mac-address 0000.0000.0002

These MAC address values were not chosen by chance: they make it easier to distinguish between entries in the table of devices connected to the factory. Let me remind you that IP addresses will not work in this case, since routing within BD disabled…

However, all these settings are still not enough for connectivity between SW1 and SW2, since ACI requires explicit permission of traffic between hosts in the form of contracts. Let’s create an object that allows ICMP packets in both directions:

Figure 9. Using a filter as part of a contract

Note the scope of the contract: it describes connectivity within AP instead of VRF. While this doesn’t make much sense in the case of only two hosts, limiting the scope of the contract generally avoids interesting special effects in the form of non-obvious connectivity between EPGs within the same VRF. All that remains is to assign the contract to the relevant EPG:

Figure 12. Contract topology when external L2 segment is presented by EPG

Time to check connectivity between SW1 and SW2:

SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
36 bytes from 192.168.0.1: Destination Host Unreachable
Request 0 timed out
36 bytes from 192.168.0.1: Destination Host Unreachable
Request 1 timed out
64 bytes from 192.168.0.2: icmp_seq=2 ttl=254 time=2.259 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=254 time=2.061 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=254 time=2.138 ms

Quite a simple process, isn’t it? Creating L2Out instead of EPG SW2 should take a little less time, since most of the settings remain unchanged. First, you need to remove the static binding of the physical port to the EPG, and then you can use the freed port in L2Out:

I was unable to convince L2Out to accept frames in VLAN 1: ACI refused to recognize the connected host, regardless of whether traffic came with a tag or without it. There were no errors in the corresponding section either, so I’m not sure if there is a limitation on the use of VLAN 1 in L2Out. Anyway, I redid SW2 settings to use SVI and connect to ACI using trunk:

SW2(config)# interface ethernet1/46
SW2(config-if)# switchport
SW2(config-if)# switchport mode trunk
SW2(config-vlan)# interface vlan 150
SW2(config-if)# mac-address 0000.0000.0002
SW2(config-if)# ip address 192.168.0.2/24
SW2(config-if)# no shutdown

Finally, the EPG must be associated with an L2 External Domain, otherwise the policies will not be enforced in the fabric.

Since the allowed connectivity in ACI is described as contracts, all external L2 entities must be classified as external EPG.

The EPG SW1 settings remain the same, so it is enough to assign the EPG SW2 contract to allow connectivity between them:

Figure 16. Applying a contract to an external EPG

Figure 17. Contract topology when the outer L2 segment is represented by L2Out

Let’s check the connectivity between SW1 and SW2 again:

SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

Something clearly went wrong. You can verify that the problem is with the contract by disabling policy enforcement at the VRF (policy enforcement) level. If the contract does indeed limit connectivity, the ICMP packets should succeed because the contract implementation is disabled. If the connectivity does not appear, then the error crept into the L2Out settings.

SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=254 time=1.713 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=254 time=1.457 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=254 time=1.397 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=254 time=1.415 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=254 time=1.548 ms

Now it becomes obvious that the problem is in the contract. However, in the first case (EPG instead of L2Out) everything worked correctly, which means that the filtering is configured correctly. What’s the catch? As usual, the devil is in the details. Of course, the contract filters are in order, as well as its application in the EPG. Now it seems to me that in this design there is only one place where you can make a mistake; however, at that time it took me several hours to find it, after which I began to go through all the possible settings within the tenant, trying to find out what exactly blocks ICMP between SW1 and SW2. It is also funny that the number of registered errors was equal to zero, i.e. there were no contradictions in the object model.

Remember the scope of the contract? We set it up as “Application Profile”, but which AP can be attributed to L2Out? Unfortunately, I have no answer to this question; however, there is a feeling that for L2Out this area of contact is unnatural, so let’s change it to a wider one, for example, to “VRF”, and check the connectivity:

SW1# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=254 time=1.807 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=254 time=1.528 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=254 time=1.412 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=254 time=1.446 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=254 time=1.391 ms

So, we have achieved connectivity via L2Out. Is it easier than setting up an EPG? More likely no than yes: additional actions were required (additional domain, EPG association with the domain), which led to a less flexible design (only one VLAN within one L2Out). Finding documentation on L2Out also turned out to be a difficult quest; information had to be collected from various blogs and forums. However, there is also a positive result: I came across curious bookwhich can come in handy when debugging ACI.

What about L3Out, you ask? As you might guess, it has the same behavior as L2Out: limiting the contact’s area of action to “Application profile” manifests itself in the lack of connectivity of external hosts with those that are connected directly to the ACI.

Thanks for the review: Anastasia Kuraleva.