How to solve complex (technical) problems

Worldview

• No hard problems. There is simply no information on how the system works
• Remember that the error occurs for a logical reason
• Be unreasonably confident in your ability to correct a mistake.
• The harder the bug you fix, the better you will be.
• Every mistake is an opportunity to learn something new.

Finding the root cause

• Try to reproduce the problem
  • Can you reproduce this from the command line?
    • It is easier for other people to reproduce the problem.
    • Easier to test the fix
• Are there any log files? What is the error message?
  • Read the error description. His every word. Twice.
  • Is there a typo somewhere (command line / config / code)?
• Isolate the problem
  • Remove some parts of the system and try to reproduce the error
  • Change one at a time, keeping everything else constant

Are you still experiencing issues? Check list

• Try to solve complex problems in the morning with a fresh mind and without distraction (solve it before you view: mail, chat, ticket system, monitoring)
• Are you having multiple problems? Try to solve the main problem first (for example ssh connection that drops every minute)
• Is this really a problem or just a misunderstanding (does it work as expected?)
  • Is there a security feature / policy that blocks your work?
• Find a stable debugging environment
• Does the problem only occur on one server? Does the same work somewhere else?
  • What’s the difference? Check it out!
• When did the problem first occur? What changed?
• Can you increase the logs log?
• Do some sanity checks
  • Are you on that virtual machine?
  • Can you ping the host?
  • Is DNS still working?
  • Check network traffic with ngrep / tcpdump. Do you see what you expect?
  • Is one of the discs full?
  • Are you editing the file you want?
    • Write garbage and try to compile and check syntax
  • Check monitoring system
    • Are there any problems with other customer virtual machines?
    • Are there any problems with other virtual machines running on the same hypervisor?
    • The entire data center is down?
  • Is the client authorized in the system? What does it do (check bash_history and ps -u)

After a while debugging

• Force yourself to express the problem easily and clearly so that the casual teddy bear understands it.
• Be patient and accept that things are taking longer than expected.
• Try to understand what’s going on. Not through endless trial and error
  • Is there any documentation to help you understand the system?
  • Talk to other people who know the system better than you
• Take a break (walk, exercise, take a deep breath, drink water, eat fruit)
• Back to the beginning: what’s the problem? What is the cause of the problem? What goal are you trying to achieve?
• Don’t have time and are obsessed with some unrelated details?
  • Take a different approach to solve your current problem

If you copy the code from Stackoverflow

• Don’t copy code from Stackoverflow without understanding the real problem
• Don’t copy code from Stackoverflow without understanding the proposed solution
  • If you don’t have time for this now => write down this solution (even when you have already figured out the problem)
  • If you don’t know what a command or tool does, read the documentation (https://explainshell.com)
  • Don’t copy commands / code. Write yourself

After solving the problem

1. Great job! I’m glad you didn’t give up!
2. What have you learned for yourself?
3. What assumptions were wrong?
4. How can you solve a similar problem in the future even faster?

• Russia’s first serial control system for a dual-fuel engine with functional separation of controllers
• In a modern car, there are more lines of code than …
• Free Online Courses in Automotive, Aerospace, Robotics and Engineering (50+)
• McKinsey: rethinking electronics software and architecture in automotive