Asterisks during tracing over IPsec

Recently, a client asked why he has asterisks when tracing. The trace went through an IPsec tunnel, so the short answer was: because IPsec doesn’t return an ICMP reply. But the client also had a routing problem, which he attributed to the asterisks. And all this happened on the USG Huawei firewall, which the client had to deal with for the first time. In other words, he was not satisfied with a short answer.

This is conditionally the topology that the client had. With AR3, the client ran a tracert to AR2, which was behind an IPSec tunnel between FWA and FWC, and got asterisks. The question was why does the client see * on the tracert command, and no other information?

The numbers on the left in the first column indicate the number of transitions (hops).

The asterisks are the number of ICMP requests (3 is sent by default).

If an ICMP Reply arrives at the interface, then the PC records the time in milliseconds. This is Delta time, that is, the time that has passed from the moment a message was sent to its receipt – round trip time (RTT). The IP address of the hop is the address of the router that sent the Reply.

In this case, we had three hops: the local USG firewall, the remote firewall, and AR2 (10.77.1.222).

Wireshark dump from the receiving interface Gi1/0/1 to FWA showed that 3 packets were sent, then 3 more packets, and there was no response to them. And only for the next three requests, ICMP responses from 10.77.1.222 were received.

Why were the first 6 packets not answered? Because no one sent them.

First. On the local firewall for the test, all traffic was configured to pass: commands were written on the interfaces undo security-manage-enablethe command is registered in the security-policy default action permit. That is, the local firewall could not reject ICMP replies, if there were any.

IPSec encrypted LAN traffic:

display ipsec sa

Flow source : 10.1.180.0/255.255.255.0 0/0-65535

Flow destination : 10.77.1.0/255.255.255.0 0/0-65535

This means that traffic from the network 10.1.180.0/24 to the destination network 10.77.1.0/24 must be encrypted.

Secondly, this is the TTL field in the ICMP packet: in each series of packets it is different: 1, 2 and 3:

When a router receives a packet in which the TTL is 1, it must send an ICMP Reply in response, in which it will report that the TTL in this packet has reached the minimum value and cannot be forwarded further, automatically reporting its IP.

This message is received by the PC that sends the traceroute and inserts this received information into the first line (1). If no response is received, an asterisk appears.

The firewall saw that the packet’s source and destination addresses were being attacked by its IPsec ACL and took over processing it. After that, he discovered that the TTL in this packet was equal to 1, and discarded it, without sending any response, without providing information about himself: he did not send a Time Exceeded ICMP ICMP message, but silently dropped the packet.

The other 3 next packets had a TTL of 2. This meant that the FWA firewall let them through, decreasing the TTL by one unit and sending it onward, in this case to the FWC firewall. In Wireshark, these three packets are on the outgoing FWA link:

These three packets had different sender and destination addresses. This is because the original packet was encapsulated by the ESP protocol.

When the FWC’s IPSec firewall received these packets with a TTL of 1, it dropped them too and did not send any response back. So the second line with asterisks turned out.

Only the next final three packets with TTL 3, reaching the destination address 10.77.1.222, generated an ICMP reply. AR2, having received these requests, answered them because it is the destination address of the packet, and because this packet has already been deencapsulated, that is, without ESP, in other words, a normal ICMP request.

Why are there only three transitions, and not many, because the route went through the Internet. Because the tunneled packet (original packet) did not enter the Internet. It was encapsulated in an outer packet that traversed this zone. Accordingly, the tunnel reduced the TTL by only 1, and traceroute treated the tunnel as a single hop, regardless of the number of hops the outer packet received.

—

For more details on how traceroute works – Deep dive into TTL: traceroute.

A routing problem that the client blamed on the “asterisks” but had nothing to do with them – Local network 10.77.1.0/24 is not visible behind the IPsec tunnel.