Building a Gigabit Switch on Linux

Network switches are simple devices: you receive a packet, you send a packet. Fortunately, people figured out how to make them more complicated, and they invented managed switches.

These are typically implemented by adding a web interface that configures settings and monitors parameters such as port status. More expensive switches may have access to alternative interfaces such as Telnet and serial console ports.

However, there is a second category of managed switches that is not immediately remembered: the switches that are inside consumer-grade routers. These routers are small Linux devices that have a switch chip inside, one or more ports that are internally connected to the CPU, and the rest are externalized as physical ports.

Here's an example of a device that documents this. I always thought that the configuration of these switch-connected ports was a convenient web interface abstraction, but I was surprised to learn that, thanks to DSA and the switchdev subsystem in Linux, these ports are actually fully functional “local” network ports. Unfortunately, since these are pretty much the only ports available inside integrated routers, they're pretty hard to experiment with.

What appears as a single line in the diagram is actually the connection between the router and switch SoC via the SGMII (or perhaps RGMII in this case) bus and the management bus, SMI or MDIO. Network switches have a lot of funny acronyms like this that, even when fully explained, seem confusing if you don't know what they mean.

It is simply impossible to control a standard commercial switch with this system because the required connections of the switch chip are not exposed for this purpose. There is only one option left…

Building Your Own Gigabit Network Switch

Surely there can't be anything particularly difficult about building your own network communicator? These devices sell for the price of a cup of coffee, and given the cost, they must have a high degree of integration. I don't see many DIY switches online, so I can assume that the chips for them are pretty hard to find…

Oh no, they're very easy to get. There's even a datasheet for them. So I created a new KiCad project and got to work.

I'm glad there's some sort of datasheet for this chip, as you usually won't find any for Realtek devices, but it's still pretty barebones. I decided to limit my search to devices that had schematics available for similar Realtek chips to see how to integrate them, and I studied a lot of documentation on how to implement Ethernet in a device in general.

The implementation of the chip initially seemed very complex: it requires about seven separate power lines and has many very poorly documented communication interfaces. After studying other implementations, I came to the conclusion that the easiest way to power it is to connect all the lines with close voltage ranges together, and we only need a 3.3V and 1.1V regulator.

It looks like the extra communication buses are needed for the extra ports, which I don't think I need. For the switch chip, I chose the RTL8367S. This is a very widely used five-port gigabit chip, although it's not actually five-port. It's a seven-port switch chip, where five ports have integrated PHY, and two are dedicated to CPU connections.

My plan, however, was different: although all of these CPU ports are available, there's actually nothing in the Linux switchdev subsystem that requires the CPU to be connected to these ports. Instead, I'll be connecting to port 0 of the switch with a network cable, and as far as the switchdev driver knows, there's no Ethernet PHY in between.

The next problem was the switch chip configuration: there are many configuration systems, and the datasheet does not say what system is minimally required for the device to work as a normal “dumb” switch. I will briefly describe the chip configuration options:

• There are eight pins on the chip that are read when it starts up. These pins are shared with the port LED pins, making design quite awkward. Also, switching from up to down requires connecting the LED in a different orientation.

• There is an I2C bus that can be connected to the EEPROM chip. The pins for it are shared with the SMI bus, which I need for the chip to talk to Linux. There is a pin configuration that allows you to choose between two ranges of EEPROM sizes, but there is no indication of what this setting changes.

• There is an SPI bus that supports connection of a NOR flash chip. It can store either configuration registers or firmware for the embedded 8051 core; this depends on the configuration of the contacts read at startup. The SPI bus contacts are also shared with one of the CPU network ports.

• There is also a serial port available, but from what I've read it probably doesn't do anything unless the 8051 has firmware loaded.

To figure it out, I just ordered a board and soldered the connections differently until I got it working. I added a pad for the flash chip, which ended up not being needed, and implemented jumpers for all the configuration pins. I left out all the LEDs, because making them configurable would have been pretty difficult.

Next, I started looking into how to properly implement Ethernet. There's a lot of documentation on this, and it makes it seem like Gigabit Ethernet requires precision engineering, impedance-matching boards, and the blessing of the Ethernet gods themselves. That doesn't really fit with reality, since these switches are very cheap to make and seem to work pretty well. So I decided to open up a switch to see if all those coupling capacitors and impedance-matching boards were actually used in real devices. It turns out, none of that matters.

Here's the schematic I ended up with, but it's different from my test PCB. I got it pretty much right the first time, though.

It seems that matching the asymmetry of the pairs is important here, but matching the length of the four network pairs is completely useless; this is mainly because network cables do not have the same twist rate for the four pairs, so their lengths vary greatly within the cable.

The pairs between the transformer and the RJ45 connector have their own ground plane, connected to the main ground via a capacitor. The pairs after the transformer are simply on the main board ground.

On the first version of the board, I made a mistake in that I forgot to include a capacitor connecting the branches from the transformer center point on the side of the switch to ground. This caused the Ethernet to barely work, so I had to manually cut tiny traces on the board to disconnect from ground. In my test system, there is no capacitor at all, and all the center points are isolated from ground. It seems to work in this design, but the final version includes this capacitor.

The result is a rather odd looking gigabit switch. It has four ports facing one direction, one facing the opposite direction, and is powered via a 2.54mm pin jack. I also added a pad for a USB Type-C connector to conveniently supply power without the need for DuPont cables.

Connecting the system to Linux

For my test system, I chose the PINE64 A64-lts board because its connectors are roughly in the right places for me. It's also important that it's not x86, since configuring it requires changing the device tree, which is impossible to do on a platform that doesn't have device trees.

The first thing I had to do was rebuild the kernel for the board, because in most kernels these kernel modules are simply disabled. To do this, I enabled the following options:

• CONFIG_NET_DSA for Distributed Switch Architecture system

• CONFIG_NET_DSA_TAG_RTL8_4 for tagging ports of this Realtek switch chip

• CONFIG_NET_SWITCHDEV — driver system for network switches

• CONFIG_NET_DSA_REALTEK, CONFIG_NET_DSA_REALTEK_SMI, CONFIG_NET_DSA_REALTEK_RTL8365MB for the switch chip driver itself

The trickier part was figuring out how to make it all boot. In theory, you could create a device tree overlay for it so that it boots using U-Boot. I decided not to do that and instead patch the device tree for the A64-lts board since I'm rebuilding the kernel anyway. I ended up with this device tree change:

diff --git a/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts b/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts
index 596a25907..10c1a5187 100644
--- a/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts
+++ b/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts
@@ -18,8 +18,78 @@ led {
 			gpios = <&r_pio 0 7 GPIO_ACTIVE_LOW>; /* PL7 */
 		};
 	};
+
+switch {
+	compatible = "realtek,rtl8365rb";
+	mdc-gpios = <&pio 2 5 GPIO_ACTIVE_HIGH>; // PC5
+	mdio-gpios = <&pio 2 7 GPIO_ACTIVE_HIGH>; // PC7
+	reset-gpios = <&pio 8 5 GPIO_ACTIVE_LOW>; // PH5
+	realtek,disable-leds;
+
+	mdio {
+		compatible = "realtek,smi-mdio";
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		ethphy0: ethernet-phy@0 {
+			reg = <0>;
+		};
+
+		ethphy1: ethernet-phy@1 {
+			reg = <1>;
+		};
+
+		ethphy2: ethernet-phy@2 {
+			reg = <2>;
+		};
+
+		ethphy3: ethernet-phy@3 {
+			reg = <3>;
+		};
+
+		ethphy4: ethernet-phy@4 {
+			reg = <4>;
+		};
+	};
+
+	ports {
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+			label = "cpu";
+			ethernet = <&emac>;
+		};
+
+		port@1 {
+			reg = <1>;
+			label = "lan1";
+			phy-handle = <&ethphy1>;
+		};
+
+		port@2 {
+			reg = <2>;
+			label = "lan2";
+			phy-handle = <&ethphy2>;
+		};
+
+		port@3 {
+			reg = <3>;
+			label = "lan3";
+			phy-handle = <&ethphy3>;
+		};
+
+		port@4 {
+			reg = <4>;
+			label = "lan4";
+			phy-handle = <&ethphy4>;
+		};
+	};
+};
 };
 

It loads the driver for the switch from realtek,rtl8365rbthis driver supports a wide range of Realtek switch chips, including the RTL8367S that I used in my setup. I removed the CPU ports from the example in the documentation and simply added definitions for the five regular switch ports.

The important part is in port@0 — this is the port on the back of my switch connected to the A64-lts; I mapped it to &emac that is, to a link to the computer's Ethernet port. The remaining ports are tied to the corresponding PHY in the switch chip.

The code also defines three GPIOs at the beginning, and they are tied to SDA/SCL and Reset on the switch board to make the communications work. Once the system is up and running, we get this:

1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1508 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
3 lan1@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
4 lan2@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
5 lan3@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
6 lan4@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff

There is a common device here eth0 and four interfaces for the switch ports defined in the device tree. To do anything, the interfaces usually need to be enabled first:

$ ip link set eth0 up
$ ip link set lan1 up
$ ip link set lan2 up
$ ip link set lan3 up
$ ip link set lan4 up
$ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1508 qdisc mq state UP qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
3: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
4: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
5: lan3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
6: lan4@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff

Now the switch is on and you can see that there is a cable connected to the third port. This system borrows heavily from the Linux networking system, so it “just works”(TM). Here are a couple of examples:

• If you add multiple LAN ports to a standard Linux bridge, the switchdev system will bridge those ports together in the switch chip so that Linux doesn't have to forward that traffic.

• Things like ethtool lan3 just get information about the link. A ethtool -S lan3 Returns all standard status information, including packets that have been fully processed by the switch.

Restrictions

There are some aspects that reduce the convenience of working with such a system. Firstly, it is necessary to create your own network switch or open a ready-made one to find suitable connections.

This system cannot be used on regular computers/servers because device trees are needed to configure the kernel, and most computers do not have kernel-controlled GPIO pins to connect the switch.

As far as I understand, there is no way yet to use this system with a network port on the computer side unless it is fixed; USB network interfaces do not have a device tree node descriptor that can be used to specify the connection port.

There is a chance that some of these limitations can be worked around: maybe there is some weird USB device that exposes GPIO pins; maybe there is some way to boot switchdev on a non-ARM device, but you'd have to read the documentation for that…