What’s under the hood of a neural network. Neural network from the point of view of mathematics and programming

Hello, my name is Alexander, I’m a backend developer.

Often, artificial neural networks are considered either from the point of view of mathematical models, or from the point of view of writing programs in a particular language. As in the parable of the elephant and the blind wise men.

The purpose of this publication is a comprehensive review of the structure of artificial neural networks from the point of view of both mathematics and program code. In this paper, the neural network is implemented in Python using the library tensorflow.keras. The article focuses mainly on the structure and functioning of an artificial neural network, so such stages as training, etc. it is not affected.

Despite the wide variety of variants of neural networks, they all have common features. So, all of them, just like the human brain, consist of a large number of interconnected elements of the same type – neurons that imitate the neurons of the brain. From the point of view of mathematics, in a neural network, each neuron is a function that depends on the values ​​of the input signals, the weights of these signals, and some activation function.

First of all, it is necessary to dwell on the concept of an activation function. It can be represented as any function of multiple arguments that returns a single value. This function is needed to determine at what input values ​​the neuron should turn on (activate), i.e. carry out the signal. And which ones don’t. The most commonly used functions are:

1. Linear activation function which has the form

2. Sigmoid which looks like

3. ReLu kind

In calculations, we will use the ReLu activation function with the parameter a = 1. Because the activation function has the form f(x) = xfor positive values ​​of the parameters, it can be neglected.

Let’s return to the structure of the neuron. Values ​​are fed to each input of the neuron (X), which then propagate along interneuronal connections (synapses). Synapses have one parameter – W (weight), due to which the input information changes when moving from one neuron to another.

X1 … xn is the value of the input signals, Y – output signal, W1…Wn – weight determines how much the corresponding input of the neuron affects its state.

Thus, the neuron also represents a certain function of the form

Where f is an activation function.

Thus, an artificial neural network is a linear polynomial with many parameters.

Consider an example. The dependence of Y on two parameters X1 and X2 is investigated.

In the course of some experiments, the following dependencies were obtained.

It is not difficult to see that dependence
But let’s say we don’t know. Let’s try to find the values Y training the neural network.

We will build several types of neural networks (with a different number of neurons), and then using the keras library in python we will find the necessary weights. A multilayer neural network will be used for calculations.

from tensorflow.keras.models import Sequential
model = Sequential()

To find the weights on each layer, use the command

for layer in model.layers: 
  print(layer.get_weights())

Consider two neural network schemes:

In the first case, the neural network has two layers, each layer has one neuron (the calculated weight is shown in the figure).

model.add(Dense(1, input_shape=X.shape[1:], kernel_constraint=non_neg()))
model.add(layers.Activation(keras.activations.relu))
model.add(Dense(1, activation='linear', kernel_constraint=non_neg()))

Let’s check, i.e. calculate the polynomial Y = (X1 * W11 + X2 * W12) * W21

To obtain similar data, the method is used predict

model.predict(…)

In the second case, the neural network has two layers, on the first layer there are two neurons, and on the second they are a neuron (the calculated weight is shown in the figure).

model.add(Dense(1, input_shape=X.shape[1:], kernel_constraint=non_neg()))
model.add(layers.Activation(keras.activations.relu))
model.add(Dense(1, activation='linear', kernel_constraint=non_neg()))

Compute the polynomial Y = (X1 * W11 + X2 * W12) * W21 + (X1 * W13 + X2 * W14) * W22

So, we have considered the structure of a neuron and the simplest neural networks, which, from the point of view of mathematics, are polynomials with a large number of parameters (weights). Training a neural network is just finding these parameters.