Your organizations heavily rely on reviews, audits, inspections and approvals just before production deployments. These inspections are usually conducted by people who are only remotely involved in your work, if any. Worst of all these inspection and approval authorities sometimes have incorrect understanding about your work, so you need to educate them just before you want to deploy your code in production.
In most organizations building control mechanisms is easier than building mutual trust. Therefore, adding more questions to change control forms, adding additional approval layers in the hierarchy and requiring extra lead times to understand and approve changes are typical problems with external control mechanisms.
Low trust environments of command and control cultures are doomed to live with a lot incidents which continuously repeat themselves. A vicious negative cycle emerges in these organizations because there is no clear transparency about real work performed, because there is no transparency about recurring problems, and because work is performed in large batches to keep number of painful deployments as low as it is possible. Longer lead times to deliver hurts the time-to-market competitiveness. Slow, late and non-actionable feedback to your teams makes learning from mistakes impossible.
A competent DevOps organization like yours know that: People who are closest to real work performed and its associated problems know the most about them. Therefore, having additional external control and approval mechanisms do not bring added value to your value stream. Which brings us to the largest management and leadership challenge of our time. Building high trust cultures where the work performed is transparent, where genuine respect and mutual trust are essence of daily work.
You need to shift reliance away from periodic inspections and start relying on peer reviews part of your daily work. Make your developers, operations engineers, quality assurance and information security specialists in your DevOps team constantly collaborate and review work of each other to make safe deployments possible.
It is already complex when one single team works on a project, but work can become even more complex when multiple teams work on different components of the same system. Some measures to keep you and your DevOps team on the top of your game are:
Many prominent DevOps organizations including Amazon, Netflix, Etsy and GitHub use “Pull Request” Review & Deployment Process. Here is how it works:
For high profile changes such as database schemas or changes that may impact information security of applications, the developer can also send additional “pull requests” to get feedback from other subject matter experts in your organization such as database administrators or information security specialists.
Small batch size principle ought to be used again while codes are being viewed. Otherwise reviewing code takes longer and this puts a lot of burden on reviewing engineers. When the change gets bigger, the risk of this change gets exponentially larger and the review becomes less reliable. If a change is too big and difficult to understand, the developer can be asked to split her change into multiple smaller and understandable chunks.
Everyone in your DevOps team regardless seniority and experience level must have someone review his or her own changes. Everyone should monitor commit streams, so potential conflicts can be quickly identified and solved before they cause larger issues in deployment pipeline, even worse during production deployments.
Pair programming by itself and pair programming in combination with test-driven development (TDD) (one developer writes code of application and another developers writes automated test) can enable your DevOps team to conduct peer reviews while the code is being written. Given the same team size and project requirements, it is measured and proven that pair programming takes 15% more time, but it reduces 85% of coding errors. Therefore, pair programming is an important practice you and your DevOps team should evaluate to use.
Furthermore, over-the-shoulder reviews, automated notifications of check-in steams and tools to assist code reviews and increased code quality are other alternatives you should be considering to implement.
Make every DevOps team in your organization be responsible for own quality of its deliverables. Build a culture that values continuous and high quality code reviews as much as writing high quality code.