Broadcast video via P2P network
When creating a video broadcast service, sooner or later, as the number of content consumers increases, the question of scaling and delivery arises. You will be faced with the problem of not only computing power, but also the bandwidth of your network.
Most modern solutions are based on the distribution of static video chunks via HTTP servers (DASH, HLS), since such data is easily cached, which allows scaling its distribution. In this case, existing content delivery networks (CDN) can be used.
Bottlenecks of private networks
The diagram shows how content is distributed taking into account geographical features, relieving the video service infrastructure.
However, unfortunately, CDNs are not capable of solving last mile problems. Let's imagine a situation where several users from the same local network are watching a broadcast, each accessing the EDGE server, creating a load on the network bandwidth. If there are enough of these viewers, they can overwhelm the channel and no one else will be able to view the content. Please note that in this case, each of them receives the same content, but for each separately.
Local cache servers
To solve the bottleneck problem, it is necessary to install a single broadcast recipient within the network, which will act as a relay.
You can place local cache servers on the most critical networks.
P2P
But it is obvious that it is impossible to place a cache server on all private networks in the world. However, what if only one of the participants receives the broadcast from outside the network and transmits it to the others inside? This approach is called Peer-to-Peer network or eCDN.
Solutions on the market
The eCDN approach has already proven itself well and has several successful implementations by various companies. Here are some of them:
How they work
Almost all solutions are a set of plugins for the main video player libraries (video.js, hls.js, dash.js, shaka, etc.).
The solutions are based on webrtc technology. This technology is designed for direct P2P connection between two users and is supported by all modern browsers.
How webrtc works
To connect two clients via the Internet, you need to build a route. For this task, a STUN server is used, which helps to overcome NAT and establish a route from an external port and IP address to a local port and IP address. This route is called an ICE candidate set. Next, these ICE candidates must be transmitted to the opposite peer to establish a connection; for this, as a rule, an additional signaling service is used.
Server part
The architecture may vary, and some business logic servers may combine the functionality of the following entities.
Authorizer
Authorizes users on the network and provides them with initial parameters.
Swarm server
Manages the network, owns information about all network nodes and determines its structure, makes decisions about which peer will be connected to which.
STUN
An auxiliary service for establishing connections via WebRTC, helps peers collect their ICE candidates.
Signaler
Required for exchanging messages between peers when establishing a connection via WebRTC.
Statistics collection service
Used to analyze network operation and tariffs.
Network topology – each to each
The principles underlying the architecture of the BitTorrent protocol are taken as a basis. The Swarm server tries to establish as many connections as possible from each peer to each, taking into account the limit on the number of connections to one peer (usually several dozen), and blacklisting unsuccessful connection attempts.
Algorithm
1. Registration
First of all, the new peer registers with the authorization server, providing all the necessary information and receiving the initial parameters.
2. Obtaining a list of peers
After registration, the peer receives from the server a list of candidate peers to connect to.
3. Connect to all available peers
Next, the peer tries to establish a connection with each candidate from the list using the STUN server and the signaling device.
4. Distribution of chunks on the network
Messages are exchanged and chunks are transferred between peers via the webrtc data channel.
After downloading a chunk from a CDN, similar to BitTorrent, the peer informs all connected participants about its presence in its memory.
Once notified that a chunk is available, peers request it as needed. If no one from the network has reported the presence of a chunk, the peer downloads it from the CDN. Thus, HTTP requests for chunks by the player are, if possible, replaced by webrtc requests to other network peers.
Flaws
These solutions are primarily aimed at reducing the load and costs of CDN.
Peers in the same P2P network are not necessarily on the same local network, hence the bottleneck problem is not solved.
The on-demand approach is more suitable for VOD, while live broadcasts cannot have a latency of less than 6 seconds.
As you can see above, managing such a network requires a special, highly loaded server that manages the logic of all connections in the network.
Own solution
Our solution is not intended to be integrated into third-party services and is not intended to save or extract money, but rather to improve reliability. This means that we must use computing resources efficiently and integrate into existing infrastructure.
Collecting information about peers and creating a network map for P2P is carried out by a statistics collection service, since it already has all the necessary information.
The most resource-intensive activity is calculating the network structure and defining the behavior for each peer. This is usually done by a centralized server. Imagine how many resources are required to online process networks of several thousand and sometimes tens of thousands of peers. By limiting our P2P networks to local networks, we were able to provide each peer with an up-to-date picture of their network. This made it possible to abandon the centralized server and entrust peers with making decisions about their behavior and the associated calculations.
Network topology – tree
Our approach uses a connection tree structure, which complicates implementation, but provides a higher level of control, reduces load and reduces spurious traffic on network nodes.
Algorithm
1. Sending statistics
Each viewer of the broadcast, regardless of whether he is a P2P participant, sends statistics about himself that are sufficient to participate in P2P.
2. Determining your network and the need to participate in P2P
Collection of ICE candidates
To determine which local network a potential peer is located on, you need to use a STUN server and collect a list of ICE candidates.
Determining the cache type
Sometimes there is a local cache located within the network. In such cases, there is no need to use a P2P network. To determine which cache the peer is currently behind, a special file is downloaded, from the headers of which it is clear whether the local cache gave it or not.
The data obtained when downloading this service file can be useful for other functions, for example for ABR.
3. Subscribe to a list of peers within your network
Since the peers themselves are responsible for making decisions about their own behavior, they need to have an up-to-date understanding of the network. To do this, the peer subscribes to all changes within the network.
This data is shared by all network participants and can be cached.
4. Determine whether you need to search for a source in P2P
The criteria that motivate a peer to look for a new source within the network are the presence of other participants (potential sources) and the low speed of receiving through the CDN (current source).
Presence of other peers in the network
If there are no other participants in the network, there is no point in running an algorithm to find the best candidate to join.
Speed measurement
There is no point in looking for a new source if the current one meets your needs. Speed is not used directly as such a criterion, but different indicators depend on the type of current source:
• When received from a CDN, the deviation from the target latency is evaluated.
• When receiving from P2P, the ping time is estimated (less than 10 ms – good, more than 50 ms – bad).
5. Selecting a suitable peer to connect to
When selecting candidates for connection among network peers, several criteria are used as a filter:
It's a different feast
You must exclude yourself from candidates for connection.
There are free slots
We make sure that the candidate has free slots for new connections.
We use a limit of child peers to three.
Not previously marked as bad
We check that the candidate has not previously been marked as ineligible.
Faster
We make sure that the connection speed is not lower than that of the current node.
Not deeper in the connection tree structure
Each peer should strive to be located in the connection hierarchy as close as possible to the CDN and be at the minimum available depth.
There are no parent and child peers in the chain
We exclude a candidate if it is in a chain of parent or child peers to avoid the formation of closed rings.
Not in the ring
In some cases, due to the asynchronous nature of the network tree structure, closed rings may still form for a short time; additional checks are made on this score.
The number of peers in the chain of parents is checked, if the depth exceeds the limit, a conclusion is drawn about a loop.
6. Connection
Once a peer receives a selected list of connection candidates, attempts are made to establish those connections.
Connection types
With a P2P connection, there are two sides: the relaying side and the receiving side.
Receiver
The peer receiving the stream is called the Receiver. A Receiver can have up to three active connections of different types.
• Basic connection – active connection through which the peer receives the stream.
• Backup connection – if the main connection is broken, the peer begins to receive the stream through the backup connection.
• Test connection – when connecting to a new candidate, its characteristics are measured, the results are compared with the main and backup connections; if positive, it replaces the current connection.
Relay
The peer that relays the stream is called a Relay. Having received a connection request from the receiver, the relay checks for free slots. Each relay can have no more than three active connections from receivers.
Signaling for connection
A significant load on the BL server is created by interaction between peers, so the necessary signaling for connection is reduced to a minimum.
Minimum set of data for connection
When establishing a WebRTC connection, peers exchange SDP (Session Description Protocol) records containing a description of the session, including the following data:
Session Description
Public encryption key
ICE candidates
Description of media sections
Since we control both participants in the connection, we can avoid transmitting most of this data and generate SDP using a template. Only two items can be considered unique data for a connection: the encryption key and the set of ICE candidates.
Pre-generation of certificate
Typically, the browser generates a new certificate for each WebRTC connection and places its public key in the SDP. However, you can generate a certificate in advance and use it in all subsequent connections. Thus, the public key becomes a public parameter that does not need to be transmitted via signaling.
Only ICE candidates needed
Unfortunately, you cannot use pre-generated ICE candidates because the ports to connect to are unique for each connection.
Signaling via dataChannel
Thus, the original SDP assembled from the template contains only one media section for data transmission (dataChannel). All subsequent interaction between peers, including signaling for SDP renegotiation and adding new media sections (audio, video), is carried out through this channel.
Data transfer methods
WebRTC provides the ability to stream media of various types – audio, video, data. This gives us some variability in how the stream is transmitted.
Streaming via dataChannel
This approach is based on replacing the player’s HTTP requests and redirecting them to the P2P network. The contents of chunks in the form of an ArrayBuffer are transmitted via dataChannel and used as a response to HTTP requests from the player. This allows you to seamlessly switch between sources, receiving chunks from any of the peers or from the CDN, if necessary, without the viewer noticing. However, this method has features and disadvantages. It is necessary to check the integrity and substitution of the chunk; it is necessary to have a checksum of the chunk and verify it when receiving it from P2P. There are some complications associated with ABR: peers can view the broadcast in different qualities, technically they are different videos with different chunk files. This problem can be solved in various ways, for example:
Using different networks for different quality options may result in the need to change networks when quality changes, which will result in additional costs. In addition, each additional network increases the load on the channel.
You can use chunks of the source peer in any available quality, and replace chunks in your own quality. But this can cause confusion for the user who expects a certain quality but gets something else.
You can limit the user's quality choices when retrieving data from a P2P network by using strict source peer quality.
Another problem may be ignoring local network congestion. Between two peers, video quality does not adapt to compensate for the drop in speed within the network.
Streaming through audio and video media sections
With this approach, it is necessary to renegotiate the SDP for each new media stream with new video and audio media sections. However, WebRTC takes on the rest of the work of managing data flows, including quality adaptation to eliminate network speed degradation. In this regard, the user is temporarily deprived of the ability to manually select quality when receiving data from a P2P network. The disadvantages of this approach include switching between sources through a small frieze.
Simulator
To debug the algorithm for constructing the topology of such complex networks, one cannot do without simulation. We have created a separate program that emulates the behavior of a given number of peers on the network, taking into account the specified parameters. The network can consist of one to hundreds of thousands of peers with changing parameters of speed, breaks, exits and inputs, etc.
Monitoring and analysis
Collection of statistics
For the functioning of the network, when collecting statistics from users, we rely on the following metrics:
Local network information
Availability of a local cache on the network
Depth in the connection tree
Which peers are you connected to?
Where does it receive data from (P2P or CDN)
Receipt speed
Speed of data transfer to other peers
Network visualization
For debugging and control, you cannot do without a tool that would visualize the network online.
conclusions
Video distribution over ECDN peering networks is an efficient and cost-effective infrastructure approach that can significantly reduce data costs. This method is especially useful for large-scale events such as the FIFA World Cup, where content needs to be broadcast quickly and reliably to large numbers of viewers.
In our case, the use of this technology solves a specific and highly targeted problem, optimizing the video signal transmission process and ensuring high quality streaming within local networks.
Potential
Advances in network technology lead to constant upgrades and increased capacity. However, this process occurs unevenly, which forces us to focus on the majority of existing networks. At the same time, demands for the quality and size of video content continue to grow.
The emergence and implementation of new communication standards, such as 5G and 6G, as well as satellite Internet and increased device performance, can take the relevance of P2P networks to a new level.
