Dynamic CDN for WebRTC streaming with low latency and transcoding

In the first part, we deployed a simple dynamic CDN for broadcasting WebRTC streams to two continents and made sure that the delays in such a CDN are really low, using the example of a countdown timer.

However, in addition to low latency, it is important to provide viewers with a good broadcast quality, because they pay for it. In real life, the channels between Edge servers and subscribers can be different in bandwidth and quality. For example, we publish a stream with a resolution of 720p with a bitrate of 2 Mbps, and the user plays it on an Android smartphone using a 3G connection in the area of ​​insecure reception of the signal, and the maximum resolution at which the picture will be smooth, only 360p with a bitrate of 400 Mbps .

The devices and browsers used by viewers are very different. For example, we publish a WebRTC stream using the VP8 codec from the Chrome browser on a PC, and the viewer plays the stream in Safari on the iPhone, which only supports the H264 codec. Or vice versa, we publish an RTMP stream from OBS Studio, encoding video in H264, and audio in AAC, and the client uses a Chromium-based browser, which only supports VP8 or VP9 for video and opus for sound.

You may also need to improve the quality of the original publication. For example, we distribute the stream from the IP camera in some reserve, most of the time the picture is static, the camera gives it at a frequency of 1 frame per second. At the same time, we want the viewer to play 24 frames per second. What if it is impossible to replace the camera or change its settings?

In all these cases, transcoding of the stream on the server will be required, that is, decoding of each received frame and subsequent encoding with new parameters. Moreover, the parameters that must be changed are often known only on the client side. Let's see how it is possible to provide transcoding in CDN, balancing between broadcast quality and server loads.

Transcoding: how, where and why?

Suppose we know the flow parameters that the client wants to receive. For example, the viewer started playing a stream, and the number of frame losses in WebRTC statistics tells us that the resolution and bit rate must be reduceduntil the client has switched channels. In this case, by default, the stream will be transcoded on the Edge server to which the viewer is connected.

If the client does not support the codec used when publishing the stream, you can assign transcoding to both the Edge and the Origin server.

Both that, and another can only be a temporary solution, provided that the configuration of Origin and / or Edge servers were chosen with a margin. Transcoding is always performed frame by frame, so it is very demanding on processor resources. So, one processor core is able to transcode a very small number of threads:

Resolution Bitrate, Kbps Number of threads
360p 1300 5
480p 1800 3
720p 3000 2

Even if we start the same transcoding process for all subscribers requiring the same media stream parameters, it is likely that several viewers with different parameters will select all the server’s resources.

Thus, the correct solution would be to select special servers in the CDN for transcoding tasks, and choose the server configuration based on these tasks.

Add Transcoder nodes to CDN

So, we will deploy one server in our CDN with the Transcoder role in the European and American data centers

Configuring Transcoder Servers:

  • Transcoder 1 EU

cdn_enabled = true
cdn_ip = t-eu1.flashphoner.com
cdn_point_of_entry = o-eu1.flashponer.com
cdn_nodes_resolve_ip = false
cdn_role = transcoder

  • Transcoder 1 US

cdn_enabled = true
cdn_ip = t-us1.flashphoner.com
cdn_point_of_entry = o-eu1.flashponer.com
cdn_nodes_resolve_ip = false
cdn_role = transcoder

Stream transcoding parameters should be described on Edge servers as special profiles in the file cdn_profiles.yml. As an example, consider three default profiles:

  • transcoding to 640×360 resolution, 30 frames per second, a key frame is transmitted for every 90 frames, H264 video codec using OpenH264 encoder, Opus audio codec 48 kHz

-640x360:
  audio:
    codec: opus
    rate: 48000
  video:
    width: 640
    height: 360
    gop: 90
    fps: 30
    codec: h264
    codecImpl: OPENH264

  • transcoding to resolution 1280×720, H264 video codec using OpenH264 encoder, without audio transcoding

  -720p:
  video:
    height: 720
    codec: h264
    codecImpl: OPENH264

  • transcoding to a resolution of 1280×720, 30 frames per second, a key frame is transmitted for every 90 frames, 2 Mbps bitrate, H264 video codec using an OpenH264 encoder, without audio transcoding

  -720p-2Mbps:
  video:
    height: 720
    bitrate: 2000
    gop: 90
    fps: 30
    codec: h264
    codecImpl: OPENH264

Publish stream test 720p resolution on the server o-eu1 and play this thread on e-eu1by specifying a profile in the stream name, e.g. test-640x360

The stream is transcoded!

Now we can describe a number of profiles on Edge servers, for example, -240p, -360p, -480p, and if a lot of lost frames are diagnosed on the client side according to WebRTC statistics, automatically re-request the stream with a lower resolution.

Group CDN nodes by continent

Now our Transcoder servers are equal. But what if we want to transcode streams by geography: for American viewers in America, for European viewers in Europe? This, by the way, will reduce the load on the transatlantic channels, since in this case only the original streams will go from Origin EU servers to America and vice versa, and not all transcoded versions.

In this case, in the settings of the Transcoder nodes, you must specify the group

  • Transcoder 1 EU

cdn_enabled = true
cdn_ip = t-eu1.flashphoner.com
cdn_point_of_entry = o-eu1.flashponer.com
cdn_nodes_resolve_ip = false
cdn_role = transcoder
cdn_groups = EU

  • Transcoder 1 US

cdn_enabled = true
cdn_ip = t-us1.flashphoner.com
cdn_point_of_entry = o-eu1.flashponer.com
cdn_nodes_resolve_ip = false
cdn_role = transcoder
cdn_groups = US

You must also add the group to the Edge server settings.

  • Edge 1-2 EU

cdn_groups = EU

  • Edge 1-2 US

cdn_groups = US

Restart the nodes with the new settings. Publish stream test 720p resolution on the server o-eu1play this thread on e-eu1 with transcoding

Make sure the stream is transcoded to t-eu, for this we open the statistics page http://t-eu1.flashphoner.com:8081/?action=stat and see the video encoder and decoder in the section Native resources

Moreover, on t-us1 there are no video encoders in statistics

More transcoders: load balancing

Let's say the number of viewers continues to grow, and the capacities of one Transcoder server on the continent are already not enough. Ok, add one more server

  • Transcoder 2 EU

cdn_enabled = true
cdn_ip = t-eu2.flashphoner.com
cdn_point_of_entry = o-eu1.flashponer.com
cdn_nodes_resolve_ip = false
cdn_role = transcoder
cdn_groups = EU

  • Transcoder 2 US

cdn_enabled = true
cdn_ip = t-us2.flashphoner.com
cdn_point_of_entry = o-eu1.flashponer.com
cdn_nodes_resolve_ip = false
cdn_role = transcoder
cdn_groups = US

However, now we are faced with the problem of load balancing across two transcoders. In order to prevent all threads from flowing through one server, we will limit the maximum allowable average processor load on Transcoder nodes

cdn_node_load_average_threshold = 0.95

When the average processor load, divided by the number of available cores, reaches this value, the server will stop accepting requests for transcoding new threads.

You can also limit the maximum number of simultaneously running video encoders

cdn_transcoder_video_encoders_threshold = 10000

When this amount is reached, the server will also cease to accept requests for transcoding streams, even if loading the processor still allows it.

In any case, the Transcoder server will continue to distribute to the Edge servers those streams that are already transcoded on it.

Ending follows

So, we have deployed dedicated servers for transcoding media streams in our CDN and, thus, we can provide our viewers with broadcast quality depending on the capabilities of their equipment and the quality of the channels. However, we still did not address the issue of restricting access to threads. We will consider it in the final part.

References

Low latency WebRTC streaming CDN is a Web Call Server-based content delivery network.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *