International DevOps Certification Academy™
How Do You Protect Your DevOps Deployment Pipeline?
After you and your DevOps team build a working deployment pipeline for the continuous delivery of your IT organization, by adhering the principles you have learnt before, you are now empowered to do low risk production deployments without external approvals. Now majority of your production deployments don’t need to go through a change approval process. This is because you put your reliance on proper design of your software delivery methodology, automated testing, automated monitoring and intelligent alerts from your environments instead of merely relying on external authorities. And yet, bear in mind that security and compliance is still an important duty of your DevOps team. This is not only to fulfil requirements from lawmakers, but also to take good care of your clients. Of course your DevOps team will still rely on and collaborate with security and compliance subject matter experts in your organization, and yet designing secure systems compliant within the legislation ecosystem your organization navigates should be now a daily task and habit of yours.
Type of Changes in Your Production Systems
- Standard Changes: Low risk changes, require no approvals, quick deployments because they are completely automated and logged. Updates of database look-up tables, content updates, styling changes, standard operating system, database or other external component patches are some of standard changes.
- Normal Changes: High risk changes, require approval from Change Advisory Board (CAB), CAB expects Request for Change Form (RFC) to assess changes, require long lead times because most of the times CAB members are not knowledgable enough to assess changes. They hold changes to give decisions ultimately based on their experience, intuition and bias on who requested changes.
- Urgent Changes: High risk changes, require approval from senior management. Critical errors in production systems which impact clients, fixes of problematic deployments, security patches, service restorations are some of urgent changes.
Regardless types of changes, you and your DevOps team need to document all changes in your change management and work planning systems (such as Remedy and Jira), so the work will be visible to everyone within and outside your DevOps team.
Use Your Track Record of Successful Automated Deployments History to Convert Normal Changes into Standard Changes
Only by increasing the ratio of standard changes over normal changes, you can avoid Change Advisory Board (CAB) approvals and you can enable faster flow and higher quality deployment pipeline.
You and your DevOps team now use your history of successful automated deployments to convert as many normal changes as it is possible into standard changes. You show CAB your track record of deployments, list of production incidents and prove them your automated production deployments cause none or only negligible incidents in your production systems.
Speed up Deployments of Normal Changes
For the changes which must still remain normal, automate composition process of Request for Change Form (RFC). Link all automated and non-automated test results, resolved and non-resolved incidents, monitoring records and logs from non-production systems to RFC. Try to simplify CAB’s job and provide all information they are looking for in the first version of RFC, so you speed up CAB approvals as much as it is possible.
Once approved, enable one-click deployments of normal changes. Regularly review types of normal changes with CAB, identify and define expectations which would convince them to convert normal changes into standard changes, so that you can automatically deploy majority of your changes in your DevOps organization.
Reduce Bureaucracy and Reliance on Others in Your DevOps Organization
The bigger an organization gets, the more food bureaucracy finds for itself. Bureaucracy has invisible ability to grow exponentially and live forever. Once you rely on authority and control mechanisms one level above you to approve your own job, you’ll no longer feel empowered and responsible for the outcomes of your own job. This will make the authority one level above you regularly fail.
Finally this authority would also require another authority to rely on and get audited. This downward spiral gets only longer until the next reorganization in your company. After reorganization, chances are almost 100% that a new downward spiral with slightly other form will replace your actual downward spiral.
Your company needs to remember that separation of duties, and reliance and control mechanisms outside your own team will not only slow down and reduce the quality of your software delivery flow, but also they make your organization less secure. A major US finance organization became victim of an ATM fraud due to a backdoor a software developer had built in ATM software. None of external auditors, external security experts, external compliance officers and external CAB authority figures could manage to identify the fraudulent source code deployed to production systems. This fraudulent could have been only identified within the team it was developed by deploying fundamental DevOps techniques such as inspection of code check-ins and code reviews.
In order to deliver quickly and securely, you need to reduce reliance on others and separation of duties because they would only prevent your DevOps team from taking responsibility. In your DevOps team you need to deploy other control mechanisms such as inspection of code check-ins, pair programming, peer reviews, automated testing and monitoring. If separation of duties are mandatory due to legal reasons, your built-in controls in your DevOps team will still empower auditors and compliance officers to give better, informed and faster decisions.
CONCLUSION
DevOps brings a new and dynamic dimension for compliance and information security. Where your infrastructure is code, when code makes your systems appear and disappear, and where your code is automatically deployed, it is not a trivial process for auditors to understand what is really going on. This is a new challenge, but as well as an opportunity to create and deploy leaner and smarter auditing and compliance mechanisms and officers.
In your DevOps team information security, compliance, quicker deployment pipeline are everyone’s job. It is a major goal for your DevOps team to enable compliance officers to access self-service information, logs, reports and metrics which prove high quality delivery of your DevOps team. These will simplify bureaucratic approval processes or even better to altogether remove them.
DEVOPS-CERTIFICATION.ORG
