Trading bot based on Node.js and Tinkoff API (Part 2) Clusters
Clusters?
Quite simply, in our project, a cluster is a situation on the market that is prone to repetition. Initially, there is only 1 criterion – the similarity of a combination of 4 candles. If the criterion is the same, we assign the current situation to the cluster. Now other criteria are being added to the cluster structure, but more on that later.
We have already seen the standard structure by which we collect combinations. To collect clusters, we at least need a tool for comparing one combination with another.
Comparison Algorithm
Standardization
Each candle has 4 meanings, in a combination there are 4 candles. We need to compare how much the set of 16 digits differs from each other. The first stage is preparing a combination for comparison, the price varies from instrument to instrument, so it was decided to represent each of the 16 values as a number from 0 to 1, where 0 is the minimum value of the combination and 1 is the maximum.
The basic formula for this step is:
// el - свеча
// min, max - значения всей комбинации
newС[kObj] = (el[kObj] - min) / (max - min);Standardized combination:
'combo': [
{ 'o': 0.98356, 'c': 0.51233, 'l': 0.51233, 'h': 1 },
{ 'o': 0.52603, 'c': 0.21096, 'l': 0.20822, 'h': 0.55068 },
{ 'o': 0.21096, 'c': 0.22466, 'l': 0.09589, 'h': 0.49589 },
{ 'o': 0.21918, 'c': 0.09589, 'l': 0, 'h': 0.28493 }
],Comparison
The very first version of the algorithm simply counted the% difference between each value and returned an array of 16 percent values. Then we calculated the average difference and that’s it, on the basis of this we decided whether the combination was similar or not.
The algorithm was recently redesigned, now we compare each candle with another candle by several parameters: the size of the candle and the upper lower tails, as well as the position of the candle relative to the combination. All this is easily calculated by the initial 4 parameters of the candlestick. The most important thing is the ability to adjust the maximum deviation for each parameter separately. Now the settings are set like this:
maxBD: 100, // максимальная разница размеров свечей (10%)
maxH: 200, // максимальная разница верхнего хвоста (20%)
maxL: 200, // максимальная разница нижнего хвоста (20%)
maxBP: 150, // максимальная разница в положении
maxCBP: 200 // maxBP, но без учета хвостовSometimes the tail of a candlestick can strongly move the entire combination up or down and, even if everything is very similar, the position of the candles will be very different, therefore CBP (Clear Body Position) is considered without taking into account the upper and lower tails.
As a result of the comparison, we get an object with parameters and an average value. The average will be used to calculate the overall match percentage.
Build into clusters
Let’s imagine a sandbox into which figures of different colors and shapes were randomly poured. Our task is to collect all the figures into groups. Already at this stage it is necessary to determine which groups we want to receive. For example, what would be the most similar figures in one group, or that the groups would be as large as possible. It is difficult to say which method is best for our purposes, collecting the most similar groups, the result will be – many small groups with very similar figures, on the one hand, the more similarity, the better, on the other hand, it is better if the group is larger. Going through combinations one after another, sooner or later we will add a combination to the cluster, and then create a new cluster to which the same combination would fit better. This is where it is necessary to decide – either we transfer the combination to a new cluster, or we do not touch it.
To begin with, we do the first iteration over our improvised database of JSON files.
let cur = 1;
const g_excluded = new Set();
const clusters = [];
while (cur <= last) { // last - последний файл
const prepdata = getDocVal(pathToF(cur)); // Данные из файла
for (let i = 0; i < prepdata.mLength; i++) {
const realElem1 = prepdata.main[i].candles;
const elem1 = makeMeOld(realElem1); // упращенная версия комбинации
// первая комбинация
g_excluded.add(transForEx(elem1)); // добавляем в сет
const cluster = [firstClusterEl(realElem1)];
// каждый кластер имеет главный (0) элемент
let cur2 = 1;
// Next code goes here...
}
cur++;
}
return clusters;After determining the first combination for comparison, we start searching for the second combination, inside this search we will compare the first combination with a bunch of second combinations and those that are similar to write in a cluster to the first. The g_excluded set stores all used combinations and checks whether it has been used before before each comparison.
while (true) {
let best_comb = null;
while (cur2 <= last) {
const prepdata2 = getDocVal(pathToF(cur2)); // данные из файла
for (let ii = 0; ii < prepdata2.mLength; ii++) {
const realElem2 = prepdata2.main[ii];
const elem2 = makeMeOld(realElem2.candles);
// Второая комбинация
let d;
if (getCollorsOfCombo(elem1) == getCollorsOfCombo(elem2)) {
// получаем результат сравнивания двух комбинаций
d = newDecoder(newPrepareToCompare(elem1, elem2), 1);
} else d = false;
if (d != false) { // если комбинации похожи
const cAfterCandle = realElem2.afterCandle; // свеча после комбо
if (g_excluded.has(transForEx(elem2))) continue;
if (d.is) {
best_comb = elem2; // запоминаем комбинацию
g_excluded.add(transForEx(best_comb)); // исключаем
}
}
}
cur2++;
}
if (best_comb == null) break; // если не нашли комбо
}
if (cluster.length > 1) { // если кластер не пустой - добавляем
clusters.push(cluster);
}The transForEx method is a very simple hash function, it makes a unique string from a combination to add it to the set of excluded combinations. The while (true) loop can be interrupted if the combination is not found after the entire traversal. The combination is skipped if excluded. Instead of skipping combinations, we can rewrite it into a new cluster if it suits it better, but more on that later. Object d – the results of comparing two combinations, in order to significantly reduce the number of comparisons, we first quickly check the colors of the candles in the combination, and if they match, we compare in detail.
As a result of processing the entire database, we have a file with all clusters. The method for assembling a ready-made element from above is not described and can differ greatly depending on its application, we have an array of combinations where 0 element is always the body of the cluster, it stores all the statistics on the combinations stored further in the array. This is done so that the body of the cluster can be painlessly detached from the rest of the combinations and used for real-time comparison.
Cluster example:
[
{ // тело кластера
"up": 1,
"uAv": 0,
"down": 0,
"dAv": 0,
"total": 1,
"allGPers": [32.804],
"allNPers": [],
"theCombo": [ // главная комбинация
{ "o": 44.4125, "c": 44.395, "l": 44.395, "h": 44.4175 },
{ "o": 44.3825, "c": 44.3725, "l": 44.3725, "h": 44.4 },
{ "o": 44.3775, "c": 44.375, "l": 44.375, "h": 44.3975 },
{ "o": 44.365, "c": 44.395, "l": 44.365, "h": 44.395 }
]
}, // комбинации в кластере
{
"combo": [
{ "o": 0.77249, "c": 0.50265, "l": 0.38624, "h": 1 },
{ "o": 0.48677, "c": 0.21164, "l": 0.15344, "h": 0.65608 },
{ "o": 0.2328, "c": 0.10582, "l": 0, "h": 0.44444 },
{ "o": 0.10053, "c": 0.61376, "l": 0.06349, "h": 0.66667 }
],
"extra": { // у каждой комбинации свои extra данные
"aftrO": 114.6325, // данные свечи после комбинации
"aftrC": 114.7875,
"min": 114.35,
"max": 114.8225,
"indexID": "f17i658" // используется для простого удаления комбо
}
},
...
], ...indexID is short for the combination in the primary database, in our case (f17i658) it is file 17, combination 658. If we replace the combination instead of a gap, we need that ID.
Conclusion
In the next part, we will take a closer look at some of the auxiliary functions and analyze the part of the project that is responsible for finding the cluster and displaying all the necessary data on the page.
