Our rake is the key to your success. DevOps and SQL Command Cases

Friday is the time for interesting stories. Today we invite you to listen to reports of DevOps and SQL-direction from the YuMoneyDay conference. Experts will tell you about:

  • the device of a cluster of logs, which allows us to understand what is happening with payments and transactions (as well as with components and services in general);
  • the work of data engineers in machine learning;
  • implementation and transformation of CI / CD.

We share valuable experience so that you do not make our mistakes. We hope it will be useful!

Our rake is the key to your success

Maxim Ogryzkov, Senior System Administrator
The report will focus on processing logs of several data centers with access through a single interface. Let’s discuss the reasons and consequences of the cluster upgrade. I’ll tell you about the transport of delivery of logs from different systems and environments, and where does Apache Kafka have to do with it. And also why we don’t use logstash and how to “attach” a cluster with one request in Kibana.

1:17 What the talk will be about: a cluster of logs
1:43 How do logs get into the cluster?
3:50 Why we chose Apache Kafka
5:02 Rsyslog: benefits of using
9:00 Where to store logs from different DCs?
12:08 What if the amount of data is too large?
14:00 Cluster upgrade.
20:30 Our rake and solutions
22:35 Translog
24:25 Bulk request
26:28 Opendistro-perfomance-analyzer
28:28 Index Shrink
29:49 Librdkafka
31:37 Summary: what our cluster looks like now

Data Engineers in Machine Learning

Evgeny Vinogradov, Head of Data Warehouse Development Department
A story about what industrial work on experiments in ML looks like – which problems are solved at the model level, and which are only at the data level, and how to ensure a controlled learning process.

1:40 Speaker Help
2:41 Who is involved in DS projects?
8:30 What is a Data Science Project?
14:15 Procedure in a DS project
15:42 Dataset collection process
20:26 How it works in Apache Kafka
29:10 What happens after collecting the dataset
29:21 How to choose a model?
30:40 Examples of problems that a data engineer can solve
34:38 What technologies does it all work on?
35:03 Conclusions of the report

CI / CD for data engineer: round trip

Anton Spirin, senior developer at BI
Report on the implementation of the principles of CI / CD in BI development, goals, their transformation and overcoming difficulties.

2:00 Speaker Help
2:44 Description of the problem
4:28 What is a data engineer?
5:43 CI / CD – what is the job of an engineer?
6:55 More about the stack and information systems
8:00 Starting point: where we started
10:34 The first stage of changes
15:50 Everything seems to be fine, but … the second stage of improvements
19:01 Almost demo: JenkinsFile, Pipelines
20:44 What did we get on the way out?
22:43 How long did it take? Release statistics
23:37 Our challenges and what could have been done differently. Future plans

All reports from the big IT conference YuMoneyDay… Materials on PM, testing and mobile development are on the way.

More reports

  • Financial service architecture
  • Make massive changes to microservices, automate code reviews and save the team’s nerves
  • More, more frontend

Similar Posts

Leave a Reply