DevOps Concepts: Unclogging the Pipelines

Releases had been delayed, slow, and buggy. You fought that for awhile and then adopted the magic of DevOps, which promised automated pipelines to get fast feedback. This also was supposed to give the developers the ability to easily fix production problems when they happened.

After all of that, you find yourself rushing to make quick fixes while the system keeps telling you that it’s still broken. This churn wastes development time, promises 2am work on live-site issues, and inures people to the red light.

The real problem is that the code going into the pipeline is still low quality: Garbage In.

Garbage Is Going In

CI/CD and automated tested does give fast feedback so that people can fix problems earlier. The churn and pain you are seeing is the pipeline working as intended. The problem is people still make many mistakes. And when garbage goes in, the pipeline can only do one of two things: it can clog, or it can let the garbage come out. A DevOps pipeline is more sensitive to the garbage than prior approaches, but it can’t transform garbage.

Continuous Delivery allows people to fix problems rapidly in production. This is a good thing! But garbage is easier to see when it comes out than when it goes in. When there is a lot of incoming garbage, developers stop spending 4-5 hours to look for garbage going in, and just push it through and see if garbage comes out. They begin trusting their ability to fix quickly, so they stop worrying about the initial quality. This simply amplifies the initial quality issues, increasing garbage in.

Is This Really a Problem?

The obvious question is if the garbage in the pipeline will get fixed, what does it matter if garbage is being put in it? The reality is that the pipeline never fixes anything. It simply takes what is shovelled in and automatically releases to your customers … live.

The good news is that IF the garbage fails a test, then the pipeline will automatically stop the shipment, giving the developer the ability to fix it and try again. And again. And again.

That puts us back to our original question. Isn’t that normal work? Let’s consider if you have 5,000 potential problems in a day that you’re relying on the pipeline to detect. Even with a 99% success rate, you’re still stuck with 50 customer problems that cost you business impact. These are why your developers are doing the live-site emergency repairs. Even with telemetry and automated rollback, these issues impact some customers and block your ability to ship the product at all.

The Original DevOps

Let’s look at the original inventors of DevOps. They didn’t use their pipelines to prevent quality issues, which is, unfortunately, the reason most people invest in the DevOps approach. The teams who created DevOps had already fixed their garbage, so they were not concerned about quality issues. Rather, they were working on large-scale systemic problems and determining how well the product meets customer desire.

Also, given that they had no garbage going in, they wanted to prevent backsliding. The DevOps pipeline was made highly sensitive so that if any garbage did slip in, it would force everything to halt. This was intentional to keep quality issues from creeping in; however, this doesn’t work when you are shovelling it in.

Fixing the Codebase

Their example shows us the clear solution: simply don’t put garbage in your pipeline. Given that your codebase likely has garbage, what do you do? The DevOps tool is still valuable to identify one piece of garbage at a time and to keep it clean once you fix it. However, it can’t handle the capacity it’s currently handling.

The technical solution to systemically transform garbage into clean code is Refactoring. Not just any interpretation of refactoring, but Disciplined Refactoring, which is a method for improving the developer’s usability of the code while proving it doesn’t change the behavior of the code – even without tests. Refactoring allows you to safely and cheaply turn hard-to-modify code into easy-to-modify code. This results in fewer mistakes when developers do change the code behavior: less garbage.

There are two soft starts to get into Disciplined Refactoring so that you can start unclogging your pipeline.

• Poor names
• Untestable code

If you want to experiment with eliminating poor names, then Arlo Belshee’s Naming as a Process Habit is a long proven step-by-step technique to improve names that is now offered under his company, Deep Roots. His Naming as a Process blog series outlines those techniques. This reduces your garbage because the number one source of mistakes is misleading or uninformative names.

If you want to experiment with untestable code, then start with Arlo Belshee’s Insight Loop Change Series, which is a set of three habits providing step-by-step techniques that he has successfully applied for many years. These techniques allow the developer to extract one untestable behavior at a time, figure out what it does, and then test it. This lets your pipeline detect garbage that was previously not detectable. More importantly, however, the act of pulling the behaviors apart allows the developers to change the parts separately. This reduces both the complexity of each change and the amount of garbage created.

Unfortunately, there’s no silver bullet to transform garbage to clean code. The good news is that you don’t need one. There are clear techniques for developers of any level to apply in their everyday coding work that will naturally clean the code as they work. While learning these takes time if taught through workshops and technical coaching, that time is drastically reduced with teams adopting very specific behaviors. Just like the fix, the learning itself has no magic shortcut. But if you improve one technique at a time, every developer will create clean code.

• ←Previous
• Next→