- the device of a cluster of logs, which allows us to understand what is happening with payments and transactions (as well as with components and services in general);
- the work of data engineers in machine learning;
- implementation and transformation of CI / CD.
We share valuable experience so that you do not make our mistakes. We hope it will be useful!
Our rake is the key to your success
Maxim Ogryzkov, Senior System Administrator
The report will focus on processing logs of several data centers with access through a single interface. Let’s discuss the reasons and consequences of the cluster upgrade. I’ll tell you about the transport of delivery of logs from different systems and environments, and where does Apache Kafka have to do with it. And also why we don’t use logstash and how to “attach” a cluster with one request in Kibana.
1:17 What the talk will be about: a cluster of logs
1:43 How do logs get into the cluster?
3:50 Why we chose Apache Kafka
5:02 Rsyslog: benefits of using
9:00 Where to store logs from different DCs?
12:08 What if the amount of data is too large?
14:00 Cluster upgrade.
20:30 Our rake and solutions
24:25 Bulk request
28:28 Index Shrink
31:37 Summary: what our cluster looks like now
Data Engineers in Machine Learning
Evgeny Vinogradov, Head of Data Warehouse Development Department
A story about what industrial work on experiments in ML looks like – which problems are solved at the model level, and which are only at the data level, and how to ensure a controlled learning process.
1:40 Speaker Help
2:41 Who is involved in DS projects?
8:30 What is a Data Science Project?
14:15 Procedure in a DS project
15:42 Dataset collection process
20:26 How it works in Apache Kafka
29:10 What happens after collecting the dataset
29:21 How to choose a model?
30:40 Examples of problems that a data engineer can solve
34:38 What technologies does it all work on?
35:03 Conclusions of the report
CI / CD for data engineer: round trip
Anton Spirin, senior developer at BI
Report on the implementation of the principles of CI / CD in BI development, goals, their transformation and overcoming difficulties.
2:00 Speaker Help
2:44 Description of the problem
4:28 What is a data engineer?
5:43 CI / CD – what is the job of an engineer?
6:55 More about the stack and information systems
8:00 Starting point: where we started
10:34 The first stage of changes
15:50 Everything seems to be fine, but … the second stage of improvements
19:01 Almost demo: JenkinsFile, Pipelines
20:44 What did we get on the way out?
22:43 How long did it take? Release statistics
23:37 Our challenges and what could have been done differently. Future plans
All reports from the big IT conference YuMoneyDay… Materials on PM, testing and mobile development are on the way.
- Financial service architecture
- Make massive changes to microservices, automate code reviews and save the team’s nerves
- More, more frontend