The first realization of oneself in the WEB or an attempt to make a distance learning system, part II

In the previous article of the tutorial, I described video communication standards and said that I settled on webRTC. He told how it works and told its theoretical implementation. In this article, I will describe the creation of the video chat itself and the server, as well as attach the code that will be on GitHub.

idea

When I had already decided on the video communication standard, I began to think in what programming language to implement the chat itself. Without hesitation, I chose two languages: JavaScript for the site and video chat.

JavaScript has a React library that allows you to create a website. I chose not thoughtlessly, because React can be linked to a site on Jango (python), so in the future you can make the design and it will be more beautiful and attractive. And now I had the task of making a workable video chat.

Server and chat implementation

Since I wanted to make both a website and an application, the first thing I decided to do was to implement the web version, and then transfer it to the desktop version.

Since I will have a server on Js, I immediately installed Node Js, which allows you to execute written code in this programming language. First, I created the react app itself.

npx create-react-app video-chat-webrtc

Then I started pulling in all the necessary dependencies necessary for the full and comfortable operation of the application.

cd video-chat-webrtc
npm i express socket.io socket.io-client react-router react-router-dom uuid freeice --save
npm run start

With the last line, we started our server, after which we can open it in the browser.

Then, I modified the /video-chat-webrtc/src/App.js file. Since I will have 3 paths for now that we can go to (Room, Main, NotFound404). I also made a pages folder, where there are other 3 folders: Room, Main and NotFound404, where Js files will be located, each responsible for its own page on the site.

import {BrowserRouter, Switch, Route} from 'react-router-dom';
import Room from './pages/Room';
import Main from './pages/Main';
import NotFound404 from './pages/NotFound404';

function App() {
  return (
    <BrowserRouter>
      <Switch>
        <Route exact path="/room/:id" component={Room}/>
        <Route exact path="/" component={Main}/>
        <Route component={NotFound404}/>
      </Switch>
    </BrowserRouter>
  );
}

export default App;

Then, after creating routers for the site pages, I finally thought about what port the site would hang on, whether the protocol would be secure or not, and, finally, over creating the connection logic itself and disconnecting the client. That is, I thought about creating a file in which all this will be implemented. In my project this file is server.js

const fs = require('fs');
const options = {
	key: fs.readFileSync('key.pem'),
	cert: fs.readFileSync('cert.pem')
};

const path = require('path');
const express = require('express');
const app = express();
const server = require('http').createServer(app);
const serverHttps = require('https').createServer(options, app);
const io = require('socket.io')(serverHttps);
const PORT = process.env.PORT || 3006;

server.listen(PORT, () => {
		console.log('Server Started!')
	}
)

serverHttps.listen(3010, () => {
		console.log("Https server Started!")
	}
)

As you can see, 2 servers start simultaneously: http and https. Why is that? When I launched it for the first time, when connecting to the http server, no data was sent, that is, the browser did not ask for permission to transfer media content. After surfing the Internet and looking for a solution to the problem, I realized that it was necessary to make an https server, since it transmits media content over a secure channel. Therefore, I had to make certificates. Since I was on Linux, it was not difficult to make such ones, although it turned out to be broken, because of which the browser said that the connection was not known and not secure, but I did it, so I had nothing to be afraid of.

Next, I created a connection on the client (that is, an algorithm that will connect the user to the server, and which will be on the client side). In the scr folder, I made a new socket folder where I made an index.js file.

import {io} from 'socket.io-client';

const options = {
"force new connection": true,
reconnectionAttempts: "Infinity", // avoid having user reconnect manually in order to prevent dead clients after a server restart
timeout : 10000, // before connect_error and connect_timeout are emitted.
transports : ["websocket"]
}

const socket = io('/', options);

export default socket;

Now that I’ve implemented the connection of clients to the server, I set about creating a method for displaying the rooms that will be created and displaying the media content that will be transferred from other connections. And I also described all the events that can be committed on the server. I made the actions.js file in the same directory as index.js, which is responsible for connecting to the server.

const ACTIONS = {
JOIN: 'join',
LEAVE: 'leave',
SHARE_ROOMS: 'share-rooms',
ADD_PEER: 'add-peer',
REMOVE_PEER: 'remove-peer',
RELAY_SDP: 'relay-sdp',
RELAY_ICE: 'relay-ice',
ICE_CANDIDATE: 'ice-candidate',
SESSION_DESCRIPTION: 'session-description'
};

module.exports = ACTIONS;

In the server.js file I added:

const ACTIONS = require('./src/socket/actions');

function getClientRooms() {
	const {rooms} = io.sockets.adapter;

	return Array.from(rooms.keys());
}

function shareRoomsInfo() {
	io.emit(ACTIONS.SHARE_ROOMS, {
		rooms: getClientRooms()
	})
}

io.on('connection', socket => {
	shareRoomsInfo();

	socket.on(ACTIONS.JOIN, config => {
		const {room: roomID} = config;
		const {rooms: joinedRooms} = socket;

		if (Array.from(joinedRooms).includes(roomID)) {
			return console.warn(`Already joined to ${roomID}`);
		}

		const clients = Array.from(io.sockets.adapter.rooms.get(roomID) || []);

		clients.forEach(clientID => {
			io.to(clientID).emit(ACTIONS.ADD_PEER, {
				peerID: socket.id,
				createOffer: false
			});

			socket.emit(ACTIONS.ADD_PEER, {
				peerID: clientID,
				createOffer: true,
			});
		});

		socket.join(roomID);
		shareRoomsInfo();
});

function leaveRoom() {
	const {rooms} = socket;

	Array.from(rooms)
// LEAVE ONLY CLIENT CREATED ROOM
		.forEach(roomID => {
			const clients = Array.from(io.sockets.adapter.rooms.get(roomID) || []);

			clients.forEach(clientID => {
				io.to(clientID).emit(ACTIONS.REMOVE_PEER, {
					peerID: socket.id,
				});

			socket.emit(ACTIONS.REMOVE_PEER, {
				peerID: clientID,
			});
		});

		socket.leave(roomID);
	});

	shareRoomsInfo();
}

socket.on(ACTIONS.LEAVE, leaveRoom);
socket.on('disconnecting', leaveRoom);

After adding some functionality to server.js, I started adding buttons to the site and adding logic to them. First of all, I went to the file responsible for displaying the main page. There I wrote the logic for displaying the created rooms and the button for creating a room.

import {useState, useEffect, useRef} from 'react';
import socket from '../../socket';
import ACTIONS from '../../socket/actions';
import {useHistory} from 'react-router';
import {v4} from 'uuid';

export default function Main() {
const history = useHistory();
const [rooms, updateRooms] = useState([]);
const rootNode = useRef();

useEffect(() => {
	socket.on(ACTIONS.SHARE_ROOMS, ({rooms = []} = {}) => {

	});
}, []);

return (
	<div>
		<h1>Available Rooms</h1>

		<ul>
			{rooms.map(roomID => (
				<li key={roomID}>
					{roomID}
					<button onClick={() => {
						history.push(`/room/${roomID}`);
					}}>JOIN ROOM</button>
				</li>
			))}
		</ul>

		<button onClick={() => {
			history.push(`/room/${v4()}`);
		}}>Create New Room</button>
	</div>
	);
}

I also rewrote server.js, because when the site was opened, it was shown that the room already exists, although no one created it. This is due to the fact that when entering the site, our socket is already connected to something, therefore, it was necessary to filter the list of displayed rooms on the screen. This function is in server.js.

function getClientRooms() {
	const {rooms} = io.sockets.adapter;
	return Array.from(rooms.keys()).filter(roomID => validate(roomID) && version(roomID) === 4);
}

Then I began to realize the rooms themselves. To display images, we needed hooks in which we will subscribe to all events. I created a src/hooks folder and useWebRTC.js file in it

import {useEffect, useRef, useCallback} from 'react';
import freeice from 'freeice';
import useStateWithCallback from './useStateWithCallback';
import socket from '../socket';
import ACTIONS from '../socket/actions';

export const LOCAL_VIDEO = 'LOCAL_VIDEO';


export default function useWebRTC(roomID) {
  const [clients, updateClients] = useStateWithCallback([]);

  const addNewClient = useCallback((newClient, cb) => {
    updateClients(list => {
      if (!list.includes(newClient)) {
        return [...list, newClient]
      }

      return list;
    }, cb);
  }, [clients, updateClients]);

  const peerConnections = useRef({});
  const localMediaStream = useRef(null);
  const peerMediaElements = useRef({
    [LOCAL_VIDEO]: null,
  });

  useEffect(() => {
    async function handleNewPeer({peerID, createOffer}) {
      if (peerID in peerConnections.current) {
        return console.warn(`Already connected to peer ${peerID}`);
      }

      peerConnections.current[peerID] = new RTCPeerConnection({
        iceServers: freeice(),
      });

      peerConnections.current[peerID].onicecandidate = event => {
        if (event.candidate) {
          socket.emit(ACTIONS.RELAY_ICE, {
            peerID,
            iceCandidate: event.candidate,
          });
        }
      }

      let tracksNumber = 0;
      peerConnections.current[peerID].ontrack = ({streams: [remoteStream]}) => {
        tracksNumber++

        if (tracksNumber === 2) { // video & audio tracks received
          tracksNumber = 0;
          addNewClient(peerID, () => {
            if (peerMediaElements.current[peerID]) {
              peerMediaElements.current[peerID].srcObject = remoteStream;
            } else {
              // FIX LONG RENDER IN CASE OF MANY CLIENTS
              let settled = false;
              const interval = setInterval(() => {
                if (peerMediaElements.current[peerID]) {
                  peerMediaElements.current[peerID].srcObject = remoteStream;
                  settled = true;
                }

                if (settled) {
                  clearInterval(interval);
                }
              }, 1000);
            }
          });
        }
      }

      localMediaStream.current.getTracks().forEach(track => {
        peerConnections.current[peerID].addTrack(track, localMediaStream.current);
      });

      if (createOffer) {
        const offer = await peerConnections.current[peerID].createOffer();

        await peerConnections.current[peerID].setLocalDescription(offer);

        socket.emit(ACTIONS.RELAY_SDP, {
          peerID,
          sessionDescription: offer,
        });
      }
    }

    socket.on(ACTIONS.ADD_PEER, handleNewPeer);

    return () => {
      socket.off(ACTIONS.ADD_PEER);
    }
  }, []);

  useEffect(() => {
    async function setRemoteMedia({peerID, sessionDescription: remoteDescription}) {
      await peerConnections.current[peerID]?.setRemoteDescription(
        new RTCSessionDescription(remoteDescription)
      );

      if (remoteDescription.type === 'offer') {
        const answer = await peerConnections.current[peerID].createAnswer();

        await peerConnections.current[peerID].setLocalDescription(answer);

        socket.emit(ACTIONS.RELAY_SDP, {
          peerID,
          sessionDescription: answer,
        });
      }
    }

    socket.on(ACTIONS.SESSION_DESCRIPTION, setRemoteMedia)

    return () => {
      socket.off(ACTIONS.SESSION_DESCRIPTION);
    }
  }, []);

  useEffect(() => {
    socket.on(ACTIONS.ICE_CANDIDATE, ({peerID, iceCandidate}) => {
      peerConnections.current[peerID]?.addIceCandidate(
        new RTCIceCandidate(iceCandidate)
      );
    });

    return () => {
      socket.off(ACTIONS.ICE_CANDIDATE);
    }
  }, []);

  useEffect(() => {
    const handleRemovePeer = ({peerID}) => {
      if (peerConnections.current[peerID]) {
        peerConnections.current[peerID].close();
      }

      delete peerConnections.current[peerID];
      delete peerMediaElements.current[peerID];

      updateClients(list => list.filter(c => c !== peerID));
    };

    socket.on(ACTIONS.REMOVE_PEER, handleRemovePeer);

    return () => {
      socket.off(ACTIONS.REMOVE_PEER);
    }
  }, []);

  useEffect(() => {
    async function startCapture() {
      localMediaStream.current = await navigator.mediaDevices.getUserMedia({
        audio: true,
        video: {
          width: 1280,
          height: 720,
        }
      });

      addNewClient(LOCAL_VIDEO, () => {
        const localVideoElement = peerMediaElements.current[LOCAL_VIDEO];

        if (localVideoElement) {
          localVideoElement.volume = 0;
          localVideoElement.srcObject = localMediaStream.current;
        }
      });
    }

    startCapture()
      .then(() => socket.emit(ACTIONS.JOIN, {room: roomID}))
      .catch(e => console.error('Error getting userMedia:', e));

    return () => {
      localMediaStream.current.getTracks().forEach(track => track.stop());

      socket.emit(ACTIONS.LEAVE);
    };
  }, [roomID]);

  const provideMediaRef = useCallback((id, node) => {
    peerMediaElements.current[id] = node;
  }, []);

  return {
    clients,
    provideMediaRef
  };
}

In this hook, I will store all the connections, a link to my media content and all media content received from other clients, and I will also store all the clients that are in the room. Also, when connecting a new user, you need to change peerMediaElements and be sure that the received data will be rendered. For this, I wrote another hook that will be responsible for this.

import {useEffect, useRef, useCallback} from 'react';
import freeice from 'freeice';
import useStateWithCallback from './useStateWithCallback';
import socket from '../socket';
import ACTIONS from '../socket/actions';

export const LOCAL_VIDEO = 'LOCAL_VIDEO';


export default function useWebRTC(roomID) {
  const [clients, updateClients] = useStateWithCallback([]);

  const addNewClient = useCallback((newClient, cb) => {
    updateClients(list => {
      if (!list.includes(newClient)) {
        return [...list, newClient]
      }

      return list;
    }, cb);
  }, [clients, updateClients]);

  const peerConnections = useRef({});
  const localMediaStream = useRef(null);
  const peerMediaElements = useRef({
    [LOCAL_VIDEO]: null,
  });

  useEffect(() => {
    async function handleNewPeer({peerID, createOffer}) {
      if (peerID in peerConnections.current) {
        return console.warn(`Already connected to peer ${peerID}`);
      }

      peerConnections.current[peerID] = new RTCPeerConnection({
        iceServers: freeice(),
      });

      peerConnections.current[peerID].onicecandidate = event => {
        if (event.candidate) {
          socket.emit(ACTIONS.RELAY_ICE, {
            peerID,
            iceCandidate: event.candidate,
          });
        }
      }

      let tracksNumber = 0;
      peerConnections.current[peerID].ontrack = ({streams: [remoteStream]}) => {
        tracksNumber++

        if (tracksNumber === 2) { // video & audio tracks received
          tracksNumber = 0;
          addNewClient(peerID, () => {
            if (peerMediaElements.current[peerID]) {
              peerMediaElements.current[peerID].srcObject = remoteStream;
            } else {
              // FIX LONG RENDER IN CASE OF MANY CLIENTS
              let settled = false;
              const interval = setInterval(() => {
                if (peerMediaElements.current[peerID]) {
                  peerMediaElements.current[peerID].srcObject = remoteStream;
                  settled = true;
                }

                if (settled) {
                  clearInterval(interval);
                }
              }, 1000);
            }
          });
        }
      }

      localMediaStream.current.getTracks().forEach(track => {
        peerConnections.current[peerID].addTrack(track, localMediaStream.current);
      });

      if (createOffer) {
        const offer = await peerConnections.current[peerID].createOffer();

        await peerConnections.current[peerID].setLocalDescription(offer);

        socket.emit(ACTIONS.RELAY_SDP, {
          peerID,
          sessionDescription: offer,
        });
      }
    }

    socket.on(ACTIONS.ADD_PEER, handleNewPeer);

    return () => {
      socket.off(ACTIONS.ADD_PEER);
    }
  }, []);

  useEffect(() => {
    async function setRemoteMedia({peerID, sessionDescription: remoteDescription}) {
      await peerConnections.current[peerID]?.setRemoteDescription(
        new RTCSessionDescription(remoteDescription)
      );

      if (remoteDescription.type === 'offer') {
        const answer = await peerConnections.current[peerID].createAnswer();

        await peerConnections.current[peerID].setLocalDescription(answer);

        socket.emit(ACTIONS.RELAY_SDP, {
          peerID,
          sessionDescription: answer,
        });
      }
    }

    socket.on(ACTIONS.SESSION_DESCRIPTION, setRemoteMedia)

    return () => {
      socket.off(ACTIONS.SESSION_DESCRIPTION);
    }
  }, []);

  useEffect(() => {
    socket.on(ACTIONS.ICE_CANDIDATE, ({peerID, iceCandidate}) => {
      peerConnections.current[peerID]?.addIceCandidate(
        new RTCIceCandidate(iceCandidate)
      );
    });

    return () => {
      socket.off(ACTIONS.ICE_CANDIDATE);
    }
  }, []);

  useEffect(() => {
    const handleRemovePeer = ({peerID}) => {
      if (peerConnections.current[peerID]) {
        peerConnections.current[peerID].close();
      }

      delete peerConnections.current[peerID];
      delete peerMediaElements.current[peerID];

      updateClients(list => list.filter(c => c !== peerID));
    };

    socket.on(ACTIONS.REMOVE_PEER, handleRemovePeer);

    return () => {
      socket.off(ACTIONS.REMOVE_PEER);
    }
  }, []);

  useEffect(() => {
    async function startCapture() {
      localMediaStream.current = await navigator.mediaDevices.getUserMedia({
        audio: true,
        video: {
          width: 1280,
          height: 720,
        }
      });

      addNewClient(LOCAL_VIDEO, () => {
        const localVideoElement = peerMediaElements.current[LOCAL_VIDEO];

        if (localVideoElement) {
          localVideoElement.volume = 0;
          localVideoElement.srcObject = localMediaStream.current;
        }
      });
    }

    startCapture()
      .then(() => socket.emit(ACTIONS.JOIN, {room: roomID}))
      .catch(e => console.error('Error getting userMedia:', e));

    return () => {
      localMediaStream.current.getTracks().forEach(track => track.stop());

      socket.emit(ACTIONS.LEAVE);
    };
  }, [roomID]);

  const provideMediaRef = useCallback((id, node) => {
    peerMediaElements.current[id] = node;
  }, []);

  return {
    clients,
    provideMediaRef
  };
}

Then I started to rewrite the logic of the rooms itself. There I display all users whose connections we have, and those who have agreed to the transfer of media content.

import {useParams} from 'react-router';
import useWebRTC, {LOCAL_VIDEO} from '../../hooks/useWebRTC';

function layout(clientsNumber = 1) {
  const pairs = Array.from({length: clientsNumber})
    .reduce((acc, next, index, arr) => {
      if (index % 2 === 0) {
        acc.push(arr.slice(index, index + 2));
      }

      return acc;
    }, []);

  const rowsNumber = pairs.length;
  const height = `${100 / rowsNumber}%`;

  return pairs.map((row, index, arr) => {

    if (index === arr.length - 1 && row.length === 1) {
      return [{
        width: '100%',
        height,
      }];
    }

    return row.map(() => ({
      width: '50%',
      height,
    }));
  }).flat();
}

export default function Room() {
  const {id: roomID} = useParams();
  const {clients, provideMediaRef} = useWebRTC(roomID);
  const videoLayout = layout(clients.length);

  return (
    <div style={{
      display: 'flex',
      alignItems: 'center',
      justifyContent: 'center',
      flexWrap: 'wrap',
      height: '100vh',
    }}>
      {clients.map((clientID, index) => {
        return (
          <div key={clientID} style={videoLayout[index]} id={clientID}>
            <video
              width="100%"
              height="100%"
              ref={instance => {
                provideMediaRef(clientID, instance);
              }}
              autoPlay
              playsInline
              muted={clientID === LOCAL_VIDEO}
            />
          </div>
        );
      })}
    </div>
  );
}

Each picture will be transmitted in the quality in which I indicate, which is why such a chat has a huge advantage, because it almost does not compress the image and transmits it as it was received in the answer or in the offer.

Conclusion of the second part

In conclusion, I want to say that the implementation of the video chat was the most difficult part of the project. If I didn’t write something in text format, then I will attach a link to GitHub, where the source of this project will be, so you only need to run it.

So, this was the second part, at the end of which I expect criticism from you, because it helps me improve)

https://github.com/DeverG3nt/video-chat-webrtc