We delete employee accounts daily

I'm a big fan of automated tests and a fairly disciplined test writer. Software design is extremely difficult to implement functionally correctly and even more difficult to avoid regression in the future. As Michael Feathers said, “Legacy code is all code that has no tests.”

Some things, such as server endpoints, database schemas, and UI library components, are very easy to test.

Other things are more difficult to test, such as endpoints that call third-party APIs, React web pages with complex states, and asynchronous tasks that require granular database records. At Airbnb, I found it difficult to test password reset emails because the email was sent through an outsourced service.

But such functionality still deserves testing, and for two reasons. Firstly, it is still important that it does not regress – due to their complexity, the likelihood of regression is high. Secondly, testing complex features often forces engineers to design the feature in such a way that it Can was to be tested. Introducing tests early in development can motivate the design of narrower interfaces and reduce coupling, leading to improved codebase quality in the long run.

To test or not to test?

The world is not ideal – sometimes we have time to write a feature, but not enough time to create automated tests for this feature. Why is this happening? By analogy with the inverse problem P=NP, many features are much easier to create than to test. Let's take as an example an application in React for creating a to-do list with a swipe delete function. It may take half an hour to develop an app, but it can take hours or even days to create automated UI tests that test the swipe-to-delete feature. This imbalance, as well as the haste that is always necessary for business, leads to the fact that development teams are forced to write features, but ignore adding labor-intensive tests.

How bad is the situation? If you're pragmatic, not really – sometimes it's worth increasing technical debt by building functionality without tests. The benefits of an untested feature can be high, the testing costs are high, and your resources are limited. Perhaps your team is short-staffed or it's a side hobby project. If you force yourself to add automated unit tests for your to-do list app, you may never release it to the first ten users. (However, if you're still not testing the swipe delete feature, even after reaching a million users, you're asking for trouble.)

Putting off automated tests is a slippery slope, but building a product requires practicing the art of strategically using technical debt. “Credit” allows development teams to quickly validate the correctness, and the discovered value of a feature can later be used to pay off technical debt (with interest). This is true for startups as well as teams building MVPs. If you spend too much time initially creating time-consuming tests, you may not have enough time to release, analyze, and develop further.

Creativity

Remember that the absence of automated tests does not mean a complete absence of testing. By default, no tests means that you are quietly forcing users to find bugs for you. Production traffic and careful alerting can be a poor substitute for automated testing, but it's better to set up quick rollbacks or flags while remaining vigilant. You won't retain disappointed users for long.

Instead of production traffic, it is better to use a subset of production traffic – often, to find a regression, it is enough for not all users to stumble upon a bug, but only a small part of them. May be useful here canary releases and beta cohorts. But there is still the problem of real users encountering real regression, and good monitoring is needed to understand when users are experiencing difficulties.

Instead of beta cohorts it is better to use dogfooding. Send real users a feature that you don't want to test automatically, but let those users be you. You can't just stop using your own product, and your eyes are a wonderful dynamic alert system (just remember to close them for eight hours every night).

Problems with onboarding

So, you have a spectrum of feature testing – automated tests, canary releases, beta cohorts and dogfooding. Our development team uses all these techniques, but there are still blind spots in functionality.

One of the hardest blind spots for us in testing is product onboarding. This is a convoluted process involving an OAuth loop and asynchronous loading of repository metadata from GitHub, queries to our databases, and unique UI elements not used in any other parts of the application. Despite these challenges, onboarding is extremely important and we need to test it somehow.

Classic synthetic tests turn out to be unstable due to bot recognition when logging into GitHub. Testing canary traffic doesn't help much because production users who fail onboarding rarely report it; in addition, sometimes in the logs they appear as indecisive, and not as users who are having problems. Beta cohorts rarely catch anything because they only go through onboarding once; the same is true for traditional dogfooding.

Spin the roulette

We came up with the following solution: a roulette script that randomly deletes one of our engineers’ accounts every day at 9 a.m. We don't just reset onboarding, we also delete accounts, tokens, custom filters, uploaded gifs, and so on.

Is this annoying? Certainly. Our colleagues come to work to code new features, but find out that they are logged out and need to create accounts from scratch. We were cautious at first when testing this technique, but we immediately saw all its advantages. Note: This is only our product account; developers retain access to GitHub and other company accounts.

Like most companies, we strive to create a fast and seamless onboarding experience. The best way to achieve this is to make developers suffer through the onboarding process every day. Across the entire technical product design team, each employee's account is deleted on average once a month. But the occurrence of problems with one colleague during the day turned out to be a sufficient price to motivate to find and fix problems.

Deleting employee accounts ensured dogfooding of one of the most critical and difficult surfaces to test. We caught a bunch of bugs and developed empathy for the user in one of the most traditional blind spots. I highly encourage other product teams to consider deleting employee accounts to achieve the same goals.

Restrictions

Is this solution ideal? No. Remote employees recreate accounts in an existing organization. Because of this, they may miss some aspects of the onboarding experience that users experience. when setting up a product for the first time in a new organization, for example, the problem of long initial synchronization. In the future, I'd like to explore the possibility of deleting the entire company's account once a month to increase dogfooding, although I'm concerned about how much it would interfere with morning code changes.

Can any product afford to delete employee accounts as frequently as we do? Probably not – some accounts accumulate not only configuration, but also a lot of useful user-generated content. For example, Instagram and Google Docs may not be able to take advantage of this approach. But for many services, especially those where user-generated data can survive the deletion of an individual account, this will benefit. Products like Datadog, Vercel, Hex and Superhuman can afford to delete employee accounts once a month. Yes, people will have to restore their personal dashboards and filters, but that’s the point.

Work for the future

Will we continue to permanently delete employee accounts? I suspect so. However, dogfooding onboarding is no better than automated testing: we constantly invest heavily in creating real end-to-end and unit tests. But dogfooding is quite different from automated testing, and they complement each other. Dogfooding catches unknown unknowns in a way that automated tests cannot. It develops empathy for users among developers and also creates a deep vision for the product that employees can use in future updates. If it's not easy for a developer to recover an account, will it be easy for a new user to create one?