New challenges from the world of continuous delivery

For future students of the course “QA Lead” and all those interested have prepared a translation of interesting material.

We also invite you to take part in an open lesson on the topic: “Organization of testing with different development methodologies”.

Continuous delivery and deployment (with the generic acronym CD) are far from new concepts. Ten years ago, Jez Humble and David Farley published book “Continuous Delivery”… Patrick Debois in 2009 organized conference “DevOpsDays” and created the hashtag #DevOps (also spelled devops, devOps, or Devops).

Of course, the CD concept is new to some teams. There are many courses and publications on this topic to help organizations successfully implement CD. As with any practice, once you get to grips with the basics, there will be problems that need to be addressed. In January 2020, I was lucky enough to visit a new DeliveryConfwhere students can gain deeper technical knowledge about continuous integration (CI) and continuous delivery. Based on what I learned there, which I can share from my own experience and the experience of the community, I would like to share some of the challenges that some teams may face today in matters of CD.

Testing

The DeliveryConf sessions I’ve been to (or later looked at YouTube channel) helped me see more examples of practical cases in which continuous delivery technology is changing the concept of testing.

Automating complex jobs or heavy labor is not a new idea, however Jessica Kerr’s performance opened up new perspectives for me. Below are some examples of how you can use automation to free up more of your time for important things:

Automation to reduce time to confirm or check machine availability.
Automating things like procedures, tools, getting credentials, PR verification process.
Deploy test environments in the cloud for each new change.

This results in fewer bottlenecks, less contention for shared resources, less time spent waiting for a resource to be transferred, and a faster feedback loop. Continuous delivery brings software to life in our minds, so we always know what’s going on with it.

Pipeline conversion

Value stream map Is one of the lean manufacturing methods. It is a way of visualizing how the team is delivering value to the client. A value stream map captures all critical steps in the process, focusing only on what adds value to the product. It shows the time it takes for each step in the value stream and the time spent waiting for those steps that do not add value, such as transmitting any data or information related to a stage, waiting time during decision making, waiting for release. resources such as test environments or people of certain specializations.

Steve Pereira In his presentation, he recommended mapping the value stream of your pipeline to identify bottlenecks and reduce feedback loops. His team created a map in Google Sheets for everyone on the team to look at. They rated manual steps, automatic steps, transfer times, and latencies. They performed one step at a time. In the end, they managed to reduce the time it took to launch the pipeline from four weeks to four days!

Here’s an example of what a portion of a pipeline value stream map might look like, including the time it takes to complete each step and the time to wait between steps:

Steve shared some great advice that I later heard in other sessions: when creating your process, you should not rely on what some well-known company does – stay within your context. Be sure to think about the people from customer service, marketing, sales — everyone else affected by your deployment and release process, reflected on the map. For example, your map should also include development support and reference documents early on, rather than at the very end, where these processes might block release.

Teams that have been dealing with CI / CD for some time most likely have legacy pipe plans – those that have been around for a while. They are usually poorly documented and nobody understands them from start to finish, they can even be extremely unstable. On DeliveryConf Laura Santamaria shared her concerns about pipelines obsolescence. She also recommended creating a value stream map for the pipeline using whatever documentation you can find, including anything recorded in the incident tracking system. This way you can better understand the pipeline and find ways to stabilize, improve and document it.

We need visualization to help us see what’s not working and talk about how to improve processes. Value stream mapping implementation is a proven technique that just works.

“Unconventional” software applications

Emily Gorchensky talked about continuous delivery in the context of machine learning (ML), about patterns and common problems. My ML knowledge is very superficial, but I was very interested. I had no idea that machine learning models have an expiration date.

Emily explained that data-driven systems are non-deterministic, making them difficult to test and improve. They have a complex acceptance criterion and are non-linear and can be regarded as a black box. Even small changes can have unpredictable effects in unexpected places. Borders are extremely difficult to define. She said the following: “Data-driven development is really difficult, and we all don’t know much about it right now.” In short, ML and CD still work hard together. It sounds a little discouraging, but it takes more experimentation to work on it.

Multiple teams may be required to retrieve data, build and test ML products, and technology transitions between them. This is where potential delays come from, so when it comes to production, the core insights the product carries are outdated. Solving these problems requires interdisciplinary collaboration and a large complex infrastructure.

The main takeaway I have made is that continuous delivery is relevant to a wide variety of software products. CD is increasingly used for web applications, firmware, operating system software. Applying it to other types of products, such as ML models, BI systems, video or audio digital products, is what teams are trying to integrate right before release. These types of products present quite big problems for continuous delivery, but the CD in them is not impossible. We can continue to look for ways to shift towards improving the quality of all types of products, and towards ways to monitor and learn how to use new technologies in production.

Talk less, work more

DeliveryConf’s strongest message is that you have to actually move towards the CD – at least in small steps. Focus on what’s most important to speed up your work. Experiment, visualize these experiments, measure your progress. CDs can even be embedded in legacy systems – we heard a story about how someone implemented a CD in VAX assembly appwho is already 45 years old!

I was struck by the emphasis on small steps and experimentation. We hear the same thoughts at conferences dedicated to testing and agile development. DevOps is all about eliminating silos in an organization. What happens if we work hard to eliminate silos in communities? We can learn a lot from each other. There are many DevOps virtual conferences, for example, DivOps, Failover Confand All Day DevOps… Use these opportunities for learning!

Automation isn’t the only important part of CD. Organizational culture and relationships between people with different skills and roles are also important. Again, this is the message we share in the testing and Agile communities, and it holds true for almost any activity that helps solve our customers’ problems with a software product.

See what else you can automate to improve an existing CD or create a new one. Automation ideas can come up organically when talking about pressing issues. Find ways to integrate important activities into your pipeline that can’t be automated, but still get real-time feedback. Look for tools and techniques to help reduce the risk of new changes in production.

Learn more about the course “QA Lead”…

Sign up for a webinar on the topic: “Organization of testing with different development methodologies”.

Right now, OTUS has the maximum New Year discounts for all courses. You can see the full list of courses at the link below. Also, everyone has a unique opportunity to send to the addressee gift certificate for training at OTUS…

By the way, about the “beautiful packaging” of online certificates, we we tell in this article…