Introduction to SCTP
Hello, dear readers!
SCTP is a rather specific protocol, which many probably haven't heard of. However, it is more than relevant in some areas, one of them is its use for delivering control messages in LTE networks.
I have recently joined the team developing software for base stations. I have not encountered this protocol before, so I decided to look into it in more detail and encountered the following problem – a relatively small number of sources devoted to this topic. Of course, they exist, and I will definitely list those that I had to use in this article.
In general, the main motivation is the desire to write a short introduction on a resource like habr to make it easier for beginners to get started.
Also, I would like to draw your attention to the fact that the emphasis is primarily on the moments where you can stumble if you don’t know about them.
First look
After the first (and superficial) attempt to google it, the following impression is formed about this protocol:
connection-oriented protocol
data delivery by messages
“reliable” delivery
the ability to specify ppid when sending – roughly speaking, we specify the identifier of the protocol used over sctp
ordering of messages within some internal streams (note: to avoid confusion, in the future, sctp streams will be referred to as streams, and execution streams, if mentioned, as threads). Streams allow, on the one hand, to ensure ordering of data within a stream, and on the other hand, to distribute independent data between different streams
a number of other advantages that will not be considered within the framework of the introduction
In general, this information may even be enough for a superficial impression. However, for the full use of this protocol, this information is still insufficient. I suggest digging a little deeper. For these purposes, we will use the book “UNIX Network Application Development” by W. R. Stevens. This book is devoted not only to this protocol, but it provides sufficient information on sctp.
I also looked at RFC 6458 from time to time.
Let's take a closer look
The first chapter devoted to this protocol begins with a trivial comparison of the sctp and tcp protocols. The general idea is that it is possible to relatively easily migrate from tcp to sctp if necessary (both protocols provide a one-to-one interface). However, as it is said later, this approach does not fully reveal the capabilities of sctp. For the most efficient use of the protocol, one should use a one-to-many interface.
It's worth stopping and thinking a bit at this point. The advantages of a one-to-many connection are generally obvious – one of them is the lack of overhead for servicing a large number of open sockets. Also, with libraries and ready-made solutions for sctp, things are not so good, and using a one-to-many connection can simplify our lives in some way (we'll talk about this in more detail later).
If we talk about a one-to-many interface, then the expected behavior is (for me personally) sending data without establishing a connection (after all, how will we detect the fact that a new client has connected?).
However, this is not the case. Let's consider such mechanisms of this protocol as notifications and associations.
Associations
In one-to-one mode, the socket is the association.
In one-to-many mode, multiple nodes connect to a single socket on the server side, each node can be identified by some integer association identifier.
Notifications
SCTP supports the ability to handle notification (by default, only the sctp_data_io_event notification is enabled, which allows each user data read operation to receive a sctp_sndrcvinfo structure, which allows extracting some information about the sender. In particular, as will be shown later, this will help extract information about streams on the side of the connected node).
Types of notifications:
SCTP_ASSOC_CHANGE – change of association (connection or disconnection of a node)
SCTP_PEER_ADDR_CHANGE – changing the state of one of the addresses of the connected node (in short, sctp allows providing something like a multicast connection, but this will not be considered here)
SCTP_REMOTE_ERROR – the connected node may send a notification about some error on its side
SCTP_SEND_FAILED – notifications about unsuccessful sending. As a rule, the reason is the association failure, and this notification will come next
SCTP_SHUTDOWN_EVENT – connection down notification
SCTP_PARTIAL_DELIVERY_EVENT – partial delivery notification. A large message may arrive in pieces. Let's remember this notification.
some other more specific notifications. It is worth mentioning that some of them, for example, are generally not supported by all implementations.
The notifications described above, on the one hand, answer the question of how to use this protocol in the “one-to-many” mode. On the other hand, the same notification about partial delivery suggests that this protocol may be somewhat more difficult to use than it seemed at first glance. But more on this later.
How to receive notifications? You should use the function sctp_recv_msgwhich will allow us to pass an argument msg_flagsuse it to determine that the incoming message is a notification and process it accordingly.
Difficulties (obvious and not so obvious)
After receiving the information, some difficulties in using this protocol become obvious:
the case of partial delivery needs to be handled. In LTE networks, such a scenario is unlikely, but theoretically, it is still possible. Therefore, be prepared to tinker a little with buffers and their clearing in various possible scenarios (at least when notifying about a connection break)
difficulties with integration with high-level libraries. The same boost.asio does not support work with this protocol. If there is a need to ensure integration, then you will have to do it yourself. Moreover, since the same
msg_flagscan be obtained through sctp functions, it becomes somewhat unclear how to integrate this functionality if its support is needed
At first glance, these are the only difficulties in using this protocol, but no:
the number of streams must be agreed upon. As mentioned above, the sctp_data_io_event event allows you to extract information about the sender. What can you extract? For example, the maximum number of incoming and outgoing streams. (These parameters have standard values, but they can also be configured). What happens if, for example, you use a number of streams for sending greater than the number of streams for receiving from another node? Nothing. The data simply won't go away (the details may be more complex, but in general – this is an incorrect situation).
as already mentioned, some notifications are generated for the same event, for example, a connection break while sending data will first result in a notification about the failed send, followed by a notification about the connection break (to be honest, I do not claim that this order is guaranteed. In theory, yes. Is this critical? Please, read the RFC). Not to mention that, for example, there may be a need to handle the failed send depending on the data transferred inside. Fortunately, the notification about the failed send will provide you with the unsent data itself for such needs.
notifications can also be delivered partially (in theory, honestly, I haven't encountered this yet). The same notification about unsuccessful sending contains a field with data that can exceed a certain limit, which will lead to delivery in parts.
when there is a need to deal with events, various questions may arise. For example, there is a subscription to notifications
sctp_send_failure_eventand there is a notificationsctp_send_failure_event_event
Well, and for dessert, a rather theoretical problem (but not a fact 🙂 ):
the same notification about unsuccessful sending. You send data. Sending fails. You receive a message, you read the information into the buffer. Then,
msg_flagsyou have determined that this is a notification and decided to process it. But the received data contains not only the information you sent, but also some service data, albeit in small volume. Now, attention, a question. You are using static buffers of a given size. You are absolutely sure that the data sent will not exceed this limit. What will happen if the stars align, and the size of the received data together with the service information exceeds the buffer size, and you start processing this notification? Theoretically, something very funny will happen
We poke a stick at orderliness
Let's refresh the known information about ordering – there are some streams within which the messages sent are ordered.
What else could it be?
Unordered message
Well, everything is simple here – if there is no need for a strict order of the data being sent, or this order is guaranteed somehow else, then you can mark the message as unordered.
However, the important point is the message it is still sent within some streamotherwise we wouldn't be able to properly process his partial delivery (here she goes again…)
Mixing modes for partial delivery
Here everything is interesting – in case of partial delivery the message is delivered in fragments. What will happen if at the moment of receiving the message in parts fragments of another message fit in, and what does the standard say about this? (see 8.1.20. Get or Set Fragmented Interleave (SCTP_FRAGMENT_INTERLEAVE) (in RFC 6458)
There are 3 types of mixing:
level 0. If partial delivery occurs, this message is blocked All associations
level 1 (default mode) If partial delivery occurs, this message blocks all other messages within the framework of the current association
level 2. If partial delivery occurs, then there are no blockings
What do these ordering modes affect? The higher the level, the less different associations and flows interfere with each other, but the more complex the buffer mechanism. Also, there are the following nuances:
if a one-to-one connection is used, then modes 0 and 1 are equivalent (one association per socket)
if mode 0 is used, then one buffer is enough
if mode 1 is used, then association identifiers must be tracked
If mode 2 is used, both association IDs and flow IDs must be tracked
if the streams capabilities are not particularly used (and this is possible), then mode 2 will be redundant (it is better to configure the streams configuration accordingly)
A great variety of connection configuration parameters
This protocol allows you to configure a very large number of parameters that affect its operation. Here is a short list:
number of attempts to send a packet
timeout between unsuccessful packet sendings (it is possible to set a “floating” timeout – it makes sense to somewhat distribute the load on the network at a certain point in time)
number of incoming and outgoing streams
number of connection attempts
Setting up automatic closing
and so on
Results of observations and conclusion
Let's rework the initial information taking into account what we've learned in the process:
connection-oriented protocol
Both one-to-one and one-to-many modes can be used
data delivery in general, by packagesbut in most cases messages
Packet mixing during partial delivery has several modes that affect the device and buffer handling
“reliable” delivery
the ability to specify ppid when sending – roughly speaking, we specify the identifier of the protocol used over sctp
the protocol is specific to libraries, but still, everything is not so good
ordering of messages within certain internal streams. Streams allow, on the one hand, to ensure ordering of data within a stream, and on the other hand, to distribute independent data between different streams.
messages can be sent unordered (but the message is still sent within the stream)
a number of other advantages that will not be considered within the framework of the introduction
To summarize, the bulk of problems are related to the issue of partial delivery.
That's all I have for now, thank you all for your attention.
