BLE adapter on ESP32 for Arduino

Looking at the abundance of cheap ESP32 modules, I wanted to make something useful out of them. For work, I needed a BLE adapter with a serial interface suitable for various applications, such as organizing a wireless communication channel between hardware or collecting telemetry from several devices. Well, for greater joy from the process, the Arduino platform was chosen. This article is about what happened.

Disadvantages of existing solutions

There are a lot of ready-made solutions on Ali, like JDY-08 or HM-10, but they are closed, do not allow connecting to several devices at the same time and do not provide data flow control. Another disadvantage will be discussed below.

The Importance of Data Flow Management

Any communication channel has a limited bandwidth. In the case of a BLE connection, it is quite limited and heavily depends on the reception conditions. What happens if you try to transmit more data than the communication channel can handle? It is the same as if you try to pour more water into a bathtub than it can handle – the water will spill out onto the floor and the data will be lost. If they are lost in the communication channel, the adapter can at least track which specific fragments were lost. But if the loss occurs in a serial communication channel, then it is simply impossible to control the scale of this loss. To prevent this, it is useful to use hardware flow control – RTS / CTS signals. RTS is especially important on the adapter side, since it prevents its receive buffer from overflowing. This signal should be connected to the CTS input on the other side of the serial connection. In the adapter configuration, the RTS signal is active by default when using a hardware port. You may not physically connect it if you do not need it.

How it works – data transfer via BLE

A BLE device can operate in two fundamentally different roles, acting as a peripheral or central device. The peripheral device has a rich internal structure, as shown in the figure below.

The peripheral device contains a set of services. A service is a collection of characteristics. A characteristic is actually a data buffer of up to 512 bytes. Characteristics can have descriptors that describe their properties and are also a data buffer, only smaller (Occam's razor is bored). The central device lacks all this internal structure, it can only establish a connection with the peripheral device, write data to its characteristics and subscribe to updates. The adapter has a single characteristic, through which data is exchanged. The adapter can work both as a central device and as a peripheral, or in both roles simultaneously. As a central device, it can establish a connection with up to 4 peripheral devices simultaneously (this is a limitation of the BLE stack implementation).

The meaning of the connection

Here it is useful to ask the question – what is the point of the connection between the central and peripheral devices? For example, in classic BT there is a serial SPP channel, it guarantees the integrity of the data stream and either delivers them in the stream or breaks the connection. TCP/IP connection has similar semantics. It turns out that BLE connection does not guarantee anything at all, it is simply needed to store the context of the connection between two devices. But it can also break on its own initiative. Characteristic updates can be damaged, lost and reordered in any way during transmission.

Transparent vs. Packet Transmission

And here we come to the second fundamental drawback of existing solutions – they are focused on transparent transmission of the data stream. This is certainly convenient for the user, but is it right? After all, BLE cannot guarantee the integrity of this stream. The adapter divides it into fragments, with which anything can happen in the communication channel. As a result, the data stream can be arbitrarily modified. The only way known to mankind to ensure the integrity of data when transmitted over such an unreliable channel is to divide it into fragments and add integrity checks (checksums) to them. And if you need a stream with a guarantee of integrity, then add delivery control in the right order, confirmation of delivery from the receiving side and retransmission on the transmitting side. We will not go that far in our adapter. But dividing data into fragments is provided in it. It receives them in the form of fragments on the transmitting side and delivers a fragment with preserving the boundaries to the receiving side. So the user does not need to divide the stream into fragments on his own. Last but not least, with transparent streaming it is not possible to implement data transfer to multiple connections simultaneously.

Control Protocol

To control the adapter and transfer data, it implements a simple asynchronous protocol, shown schematically in the following figure.

Any simple device with a serial port can act as a control subsystem. The adapter receives commands and data for transmission from it. In turn, it transmits data received from connected devices, notification of its status, and debug messages. There are only two commands – reset and connection of the list of devices. Devices are identified by their address. Identification by name is not provided, since it requires an additional procedure for searching for a device, and the names are not unique.

The implementation of the protocol in python is contained in the file python/ble_multi_adapter.py

Transferring binary data

Since the protocol uses certain bytes as markers for the beginning and end of a message, the presence of these bytes in the transmitted data violates the exchange protocol. To transmit arbitrary binary data, it must be encoded in base64 before sending it to the adapter. A byte with a value of 2 is added to the beginning of the data block as a marker for encoded binary data. The adapter will decode the data, send it to the receiving side, where it will be encoded in base64 again.

Advanced Batch Mode

The adapter assumes that the data packet it receives can be written to a characteristic, which limits its size. The adapter uses data fragments of up to 244 bytes, which theoretically should be transmitted without further fragmentation. To transmit larger data packets, the adapter implements an extended packet mode, in which packets are divided into fragments, each of which is equipped with a one-byte header and a three-byte checksum, as shown in the figure below.

Using the extended packet mode is completely transparent to the user and is recommended for all cases except those where interaction with other BLE devices is required that are unable to handle fragmented extended mode packets. The extended mode is enabled by default in the configuration file (EXT_FRAMES).

Error handling

The adapter uses a simple but extremely effective error handling strategy – it simply restarts. This also happens when the connection is broken.

Problems

During the work on the project, many problems and even bugs were discovered in the libraries and Arduino classes based on them.

As already noted, using serial port flow control is good and correct. But there is one nuance. When using two-way RTS/CTS control, a deadlock between the transmitter and receiver is possible when they both try to write something to the serial channel, despite the fact that both receive buffers are full. The deadlock occurs because both the transmitter and receiver can either transmit or receive data, but cannot do it at the same time. There is a seemingly simple and obvious solution – do not add data to the transmit buffer if there is no room for it. Unfortunately, the ESP32 library does not allow you to reliably know the size of free space in the transmitter buffer. So at the moment, the recommendation is to use RTS to prevent the adapter's receive buffer from overflowing, but not to use CTS.

It should be noted that when the adapter works via USB CDC, there is no flow control at all. New data simply overwrites old data.

The unexpected thing was that the BLERemoteCharacteristic::writeValue method, which is called to write data to the peripheral device, is not usable at all. According to the authors, it should wait for the previous write to complete. To do this, it takes a semaphore, but does not give it back in case of an error, so the next call hangs forever. We had to duplicate this functionality in the adapter code.

Interestingly, unconfirmed writes, which are the default, lead to unstable connections if there is more than one. Confirmed writes do not have this problem.

As it turns out, the adapter's simultaneous operation in two roles, although possible, can lead to connection instability. Perhaps this is even a feature, not a bug. The central device controls the periods of activity on the air both for itself and for the connected peripheral devices. This means that a device with two roles must simultaneously both control the periods of activity of its radio module and be controlled by another central device. Perhaps this contradiction leads to problems when using two roles simultaneously.

Compilation, settings

To compile you don't need anything except Arduino. In the settings (Additional board manager URLs) add a link to the ESP32 support package:

https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json

Download the package esp32 by Espressif Systems in the board manager. Select ESP32C3 Dev Module or ESP32S3 Dev Module depending on what processor you have on your board. It might work with others too, I tested it on these two. Allow it in the board settings USB CDC On Boot. Set the correct COM port corresponding to the connected board, and the board can be flashed. Keep in mind that boards without any firmware at all enter a reboot cycle when connected. To flash such a board, it must be put into boot mode. To do this, press and hold the BOOT button, then press and release the RST button, then release BOOT. After flashing, you need to press RST to exit boot mode.

The project has many compile-time settings that are located in the header file. mx_config.h. It includes another header file (by default default_config.h), which you can copy, rename and customize to your liking. In the settings, you can select the name of the device, its operating mode, the device address for automatic connection if you need a communication channel that is established automatically, and much more.

Performance

The maximum data transfer rate obtained in the echo test for one connection is about 4kB/sec in one direction (+ the same in the other direction). For four connections in each of them the speed drops to 1.5kB/sec, which is apparently a limitation of the serial port. If desired, its speed can be increased in the settings (UART_BAUD_RATE).

Data transfer from the peripheral to the central device during channel overload demonstrates an increase in lost packets. In fact, only as many packets as the channel can handle are delivered, which is expected since they use a fully asynchronous notification mechanism. Writing in the opposite direction from the central device to the peripheral is more resistant to overload since it uses a write with confirmation and during overload it begins to slow down the reception of data from the serial port.

Energy consumption

The ESP32C3 Super Mini module consumes about 65 mA at rest. Under load, when receiving 50 short messages per second from 3 connections, the consumption increases to 120 mA. The ESP32S3 module, as expected, consumes more due to two processor cores – about 90 mA at rest. Under load, however, the consumption does not increase so much – up to 115 mA. In general, it is normal, but not for battery power.

Communication range

Depends heavily on the antenna. The worst option is a chip antenna, like those on the ESP32C3 Super Mini modules. It converts electrical energy mainly into heat. You shouldn't expect a range of more than 10 meters from it. A printed antenna is already much better. The best results are given by external antennas, even the simplest ones. If there is a chip antenna on the board, and there is no connector for an external one, just remove the chip antenna and solder an external one, as shown in the photo

The range can be further increased by software increasing the transmitter power. This option is enabled by default in the configuration file (TX_PW_BOOST). In this version, the range in open areas is 100 m. Indoors, stable operation is possible between adjacent floors through a reinforced concrete floor.

Compatibility

The adapter can work together with JDY-08 and similar adapters, provided that you transmit no more than 20 bytes of data at a time. It is also fully compatible with Web BLE applications, such as thisThe application at the link is convenient to use for testing.

Source code

It lies here https://github.com/olegv142/esp32-ble

Project directory for Arduino ble_uart_mx