The true value of continuous delivery is that high-quality, working code is ready to deploy to production, on-demand, at all times through an automated delivery pipeline.

As code is merged up to the production branch, it is tested through automation from Development Unit Test and TDD through BDD and business testing in increasingly production-like environments. As this code progresses through the software development lifecycle (SDLC) and tests pass, it is ready to deploy when the business requests the change. Importantly, the risks associated with going live decrease as confidence builds that the software will work in production and necessary monitoring is already in place and tested prior to delivery.

Hence, continuous delivery makes teams more efficient and gives the business the flexibility to decide when functionality should be switched on.

 

Business benefits 

The business benefits from realising return on investment consistently, continually, and safely. Businesses have the control and ability to decide when to turn on functionality to suit market conditions and customer requirements, transforming customer perception of business delivery. Internally, production costs are lowered through reduced development time and effort, increased output efficiency, stability and speed to market.

Other internal benefits include:

• streamlining workflows,
• lower staffing costs,
• reduced attrition rates,
• retention of domain knowledge,
• improvement in operation confidence through enhanced and collaborative teamwork.

Through our strategic partnership with Google Cloud, Deutsche Bank has been assisted in implementing objectives and key results to measure defined business benefits for our product releases. Are we building what the customer wants? Are we building it the right way so it’s both maintainable and performant? To retain and grow customer loyalty, and therefore revenue, companies need to ensure that their applications are performant, usable, and attractive to the end-user. Pre-production must include confirmation of resiliency, efficient disaster recovery, and security.

Continuous delivery defines the ability of companies to release changes on demand, safely and sustainably, even during normal business hours. This includes, but is not limited to updating services in a complex distributed system, upgrading mainframe software, making infrastructure configuration changes, making database schema changes, and updating firmware automatically.

Within Deutsche Bank, a huge amount of work has been put into prioritizing our DevOps solution, to facilitate Continuous Delivery. Within Deutsche, the Group Agile Accelerator Team has spent much of 2020 working on helping teams working efficiently and effectively with both Agile and DevOps processes and tooling. For 2021, the focus has now switched to the Business ROI via Continuous Delivery. This has been a fundamental shift for the bank, where application teams put forward a business case for funding with defined business value.

Advancement of DevOps in particular was the key focus to ensure the ability to continuously deliver is a highly achievable and crucial game-changer, where financial backing through the business now sees a BOW that contains technological advancements in addition to new features. Starting with business value and funding and understanding what would be needed to attain this, this shift in team culture and focus, empower teams and further unites Business, Development, infrastructure, and Operations.

This change in approach and having specific funding has enabled Business to understand the true drivers and benefits for continuous delivery and, of course, bears witness to the successful DevOps and Agile transformations that Business have already reaped benefits from.  Funding has also delivered a highly motivated and exploratory Community, bringing all sectors of the Bank together, targeted at sharing and learning in a cohesive way of working, which in turn is reducing costs through high quality and frequent stable releases by teams working in partnership with Business.  Budgeting for 2021, therefore, saw Business and the Executive Board strongly support the transformation across the bank.

 

The challenges

A common mistake organizations make, not only with continuous delivery but also with Agile and DevOps implementation, is they take someone else’s solution, emulate it and enforce it.  This often stifles innovation and team empowerment as they are given no opportunity to develop and evolve their own way of working. Whilst I would encourage organizations to understand failure points and challenges for companies who have implemented a successful continuous delivery strategy, they must fully understand their own issues and infrastructure and technological roadblocks. Value Stream Mapping is particularly beneficial and a great starting point so long as it involves all necessary stakeholders.

One of the greatest challenges to continuous delivery is related to the existing architecture and infrastructure in use. At Deutsche Bank, applications built in the last few years have been containerized and decoupled, built with automation being accessible and exposed APIs both for Test and Deployment ease. The challenge is more around the older monolithic applications. It is often not possible to automate deployment and testing remains largely manual or partially manual.

Some organizations mistakenly believe that they can implement continuous delivery by doing their existing deployment process more often. However, implementing the technical capabilities that drive continuous delivery typically requires significant process and architectural changes. Increasing the frequency of deployments without improving processes and architecture is likely to lead to higher failure rates and burned-out teams. Using modern tooling without implementing the necessary technical practices and process change won’t produce the expected benefits.

Together with Google, we are have implemented DevOps Research and Assessments (DORA) assessments for application teams. Through DORA research, Google provides interactive tooling which enables applications to assess their maturity in comparison with other companies in the same field. It is this type of advance in processes such as DevOps that enables companies on an ongoing basis to ensure the pipeline facilitating continuous delivery continues to improve and thereby protect the integrity and delivery capacity of production

Finally, implementing an efficient continuous delivery process will ultimately fail if the right KPIs are not agreed upon, transparent, and continually reviewed and upgraded.

Metrics may include:

• Defect arrival rate, number of failed tests, and at what point in the SDLC
• Number of builds ready for deployment and number of failed builds
• Number of daily deployments to production
• Deep insights into CD pipeline execution
• The health of Applications post-deployment
• Lead time between code commit and production release
• Mean time between build failures, defect resolution, and recovery (MTTR)
• Production downtime and recovery during and after deployments

 

 Implementing continuous delivery

Ideally, continuous delivery is implemented through a fully stacked deployment process, with small iterative high-quality releases deployed frequently. This involves a collaborative and transparent methodology, preferably Agile Design and Delivery, with a decoupled architecture using key tools. Quality Assurance must be a focus through the entire SDLC, ideally utilizing TDD and discussion through collaboration with test Professionals, Developers, and Business representatives. Company culture must shift to being business benefit-driven, minimum viable product iterative releases reducing risk.  These “3 Amigos” should include working with Security, Operations, and Infrastructure to ensure seamless delivery to Production, only if all these groups become part of the Feature Delivery Team can a successful delivery be achieved.

Through our partnership with Google and our continuing transformation towards an Engineering culture, Deutsche Bank has brought in a number of changes to both measure DevOps Maturity, making radical shifts in how DevOps is funded and deployment changes.

Our focus on DevOps across the many thousands of Applications we have is to introduce the DevOps Research and Assessment (DORA), as referenced in the question above, DevOps Maturity tool provided by Google.  There are a couple of options with this tool, firstly there is a quick assessment, this is available online and enables teams to get a high-level understanding of their week areas and where teams need to focus.  It also provides a cross-analysis against other companies in the same Sector.

A key reason to utilize DORA is that it provides us an efficient and retrospective quantifiable methodology to evaluate if the goals of continuous delivery are being realized.

Before implementing a continuous delivery pipeline and to avoid failure there are a number of precursors that teams need to be able to demonstrate:

• Is our software in a deployable state throughout its lifecycle?
• Do we prioritize keeping the software deployable over working on new features?
• Is fast feedback on the quality and deployability of the system we are working on available to everyone on the team?
• When we get feedback that the system is not deployable (such as failing builds or tests), do we make fixing these issues our highest priority?
• Can we deploy our system to production, or end-users, at any time, on-demand?

A final implementation key criteria are that the release lifecycle does not end with release to production but encompasses operating and monitoring in production. This means incorporating all user feedback, incidents, monitoring reports, usage statistics, and so on into the planning phase of the next release.

 

Who is in charge of putting and overseeing a successful continuous delivery strategy?

There is no one person that is responsible or can be responsible for a continuous delivery strategy. Multiple disciplines must be involved in defining and governing the continuous delivery process, Business, Product Owners, the team including architects, developers, QA, Operations, Security and Release management.

In industries such as banking, it is often feared that a continuous delivery pipeline is not possible in such a highly regulated environment. In reality, this is not true. The pipeline, if built correctly, should have automated governance built-in. Indeed I would argue more governance than other methodologies.

For example, requirements are discussed and understood at a far better level both in terms of technical implementation and how they will be supported in production. Delivery is quick, typically bite-size implementations following two-week iterations, using SAFe (Scaled Agile Framework) for example, organizations are able to set out a framework with a collection of methods and rules that facilitate organizations to scale delivery at an enterprise level. Delivery of deployment code to production is done regularly and on-demand. The release cycle has become a rigorous and highly tested rapid one, for example, building on the concept of Program Increment which is the foundation of SAFe.

Quality gates are defined in very different ways. Stories hold highly challenged and discussed Acceptance Criteria and cannot be merged into the production branch until all of these are met, and it has been proved they have been met through demos and passing automation. Quality thresholds around code quality, design, reuse, and avoidance of “code smells” are inherent in the delivery pipeline through code review, either peer or through tools such as SONAR.

What has changed significantly is the decision to release (or not) has undergone a major transformation. The right process, methods, tools, and reports must form an integral part of the continuous delivery strategy. Within Deutsche Bank, we have transitioned to an Agile release Template that governs releases. The process must include the right measures post-production to ensure the strategy is working. If something breaks in production, not only is failure fast key to quick resolution but the process itself must be scrutinized and adapted as the highest priority. The right strategy empowers teams to make the release decision themselves as releases become more predictable and failure rates/incidents become a seldom seen problem and where they are observed can be resolved in a matter of hours.

A continuous delivery strategy must include transparency in measurement and monitoring of high service availability, delivery of critical fixes within hours, and deep analysis and monitoring of customer usage. You also need the right people in the right DevOps roles with the right skills, and at the top of this is having the right key soft skills and the willingness to collaborate. A learning culture must be fostered where it is safe to fail, fail fast, fix it and learn from the failure.

Release engineers typically replace the traditional and often bureaucratic role of the release manager. Again, this involves team members with a mixture of skills.  Within Deutsche Bank, Production Engineers (PEs) are evolving to fulfill this part of the process, focused on technical details, bringing together coordination of the release from development through to production and beyond.

A further key role in the continuous delivery pipeline is the architect. With a focus on ensuring high availability on production and pre-production systems, Architects are key to the design implementation, for coding, testing, and deployment. Through our partnership with Google and our increasing transition towards cloud platforms, Architects have performed a critical role in ensuring application design is correctly built right, the first time and lends itself to Cloud deployment and all the benefits associated with this.

Of course, automation has to be central to any continuous delivery strategy, whether for building an automated testing framework or an automated deployment pipeline. Developers, testers, and Operations are no longer three distinct teams. Developers that embrace the testing mindset and work with QA members to build code originating bypassing the identified test through TDD will ensure a substantial shift left, a massive reduction in late defect detection and corresponding cost, and a continued collaborative review that the code is focused entirely on Business criteria by meeting the Definition of Done.

Likewise, the next level of testing BDD builds on the Unit Test framework already developed in partnership and continues to the notion of building tests that verify Business objectives, BDD should never be about automating traditional black-box tests but be scenario-driven with the right mix of negative and positive paths.

Similarly, security engineers work with the team throughout the SDLC.  Non Functional aspects are now identified within Acceptance criteria and form part of the definition of done for a particular feature.  Security, Performance, Maintainability, and Sustainability, etc. are now integral parts of the design rather than late considerations often when it is too late to provide solutions or even worse not considered until Production Incidents evidence that this has been missed from the design considerations.

Operations and infrastructure engineers being brought in as part of the team is the other critical part of delivering a strategy that is entirely geared to delivering working software that meets business requirements in a safe and fast way. Through improved environments that are increasingly production-like, a better understanding of the infrastructure, environment, scalability, and how the system operates delivery of or turning a feature on in production becomes infinitely safer. Reliability, disaster recovery, security, usability at scale, and robustness of an application can be fully assessed and tweaked to be production-ready ahead of time.

Aside from the people focus there are well-acknowledged and defined criteria that a continuous delivery strategy must state process and policy:

• Test automation
• Deployment Automation
• A solid code Merging policy held within a repository where branches and forks have a very short lifespan with frequent merges into the main branch
• Integration of security into feature design and development
• A loosely coupled architecture
• Code maintainability
• Empowerment of teams to choose the right tools for them and to define the right strategy for their Application
• Continuous integration
• Continuous Testing
• Version control
• Test data management
• Monitoring and observability against predefined metrics and criteria including proactive notifications to teams
• Database change management

 

You cannot be successfully Agile without Continuous Delivery!

You cannot be successfully Agile without Continuous Delivery, in terms of delivering business value quickly and safely to production and achieving business benefit. You can be the most efficient Agile team in the world but unless you have an automated delivery pipeline you just hit a new roadblock. At a most simplistic view, Agile, DevOps, and continuous delivery must be seen as the working processes that drive a means to achieve business delivery. Business requirements become the Agile stories that are developed and tested throughout the SDLC and verified against business acceptance criteria.

DevOps provides tooling and automated solutions to transition these deliverables to production through a collaborative and iterative way of working between all involved teams. Continuous delivery is the mechanism for turning these business requirements into working software that provides the right solution as already demoed to the business throughout this pipeline. Operations, Development, and Infrastructure must have transitioned through the cultural shift of working as one team with one combined book of work through all stages of this process and beyond release into production.

However, the two, in reality, must go hand in hand, CD needs small iterative releases to ensure releases become non-events, smooth, risk-averse, and efficient. At its essence, one of the inherent guidelines of a successful Agile team is that the team is empowered. The traditional release management process is a prime example of an organization being anti-agile, it places zero trust in the team and their ability to release high-quality software to production.

Another key anti-agile feature is that those that “govern” the releases are often outside of domain knowledge, their policing is around, has testing been done, has code been reviewed, has disaster recovery been proved, etc. yet there are no questions around OKR’s, does it provide Business Value.  The level of bureaucracy and layers of approvals is oppressive to delivery teams and cripples the ability to release.

In addition to Agile working in partnership with CD, other factors such as automated testing and automated deployment are pivotal parts of the implementation. Without this, organizations continue to be hampered by exhaustive manual testing and error-prone and lengthy deployment meaning business value cannot be delivered in a timely manner. Such organizations also miss the fast feedback loops, fast failure, continuous learning inherent in an Agile DevOps methodology.

At Deutsche Bank, part of our Agile methodology has included the delivery of a QBR (Quarterly Business Review). This has been a massive and highly successful transition for the Bank, again this has been implemented and taught from the GAA. This symbolizes the huge shift into consistently focusing and testing business values. Gone are hugely technical lengthy specifications which cannot be understood by Business. Agile teams are entirely focused on ensuring they can represent the actual business benefit they are delivering, what the usability is, who the target audience is, what the goals are and how they will be measured. Teams have become merged with business, these are no longer polar opposites they are one team that constantly challenges and supports each other in a transparent and entirely comprehensive manner.

In summary, companies that continue to purely focus on implementing Agile or implanting continuous delivery pipelines will continue to fail to deliver Business Value.  Organizations must view the Agile and DevOps not as two distinct parts but as a collective partnership in transitioning the culture and the method of development and delivery to Production or they will continue to inherently stifling any benefits of either.

 

Should every business adopt continuous delivery?

It depends on the challenges. Companies should use a checklist to assist with prioritising continuous delivery implementation, it’s not just about building the right thing it’s about building it in the right way, at the right time. For example:

• The frequency of a process or the number of times it is repeated
• Elapsed time
• People and resource dependencies and potential roadblocks
• How much would the process benefit from automation, is it error-prone
• What is the urgency of getting this process automated against other processes

There are a number of factors that must be in place before an organization considers transitioning to CD as a way of delivering software speedily, safely, and sustainably.  So whilst the vast majority of companies would benefit in the business value driven through CD organisations need to assess is the company in its entirety committed.

At an organization level companies should grade themselves against the following considerations:

• Is the company ready and open to adapt to a new culture?
• Is interoperability between technologies an issue?
• Does the company have complex monolithic applications?
• Does the company work in an Agile way to deliver small incremental functionality (CI)?

Organizations who feel confident they are at the right place to move to CD should review Applications individually against criteria such as the following:

• Does the Application have a high level of automated testing both pre-implementation and post-implementation?
• Is environment provisioning automated and tested on production-like environments with osculated Production data?
• Is the Application DR compliant?
• Are Production Change Management approval groups in place this includes Technical Operations, Business, and QA Test Manager Groups, and has the approval process been automated??
• Is the Application already releasing into production frequently?
• Does the Application have a good history of quality releases, with no critical or high defects relating to application releases?
• Does the Application have evidence of Issue closure success in terms of Time to remediate (MTTR)?
• Are Deployment (and Rollback) activities largely, if not fully, automated? There should be no resource dependencies on SAs/DBAs to perform manual implementation activities.
• Is a central shared code repository set up for the Application, which is tested and validated continuously?
• Are Developers integrating code to the Main Branch several times a day through a defined Continuous Integration process?
• Has Infrastructure been built into the code repository?
• Has Monitoring been built-in throughout the SDLC and tested that the right alerts are being triggered?
• Is code quality reviewed, poor code, complexity, duplication, and other “code smells” prevented from being merged to the Production Branch?
• Is Deployment automated, this must not be confused by developers writing automated scripts for creating deployment pipelines that require significant and frequent manual updates which are error-prone?

 

Conclusion

Implementing continuous delivery as part of a company’s DevOps process is essential.

Agile and DevOps without continuous delivery mean that the key business value of having deployable release-ready software at any time is not possible.  As mentioned before Deutsche Bank’s investment in the “DevOps booster” saw a significant shift in the organizational culture from technical and business teams with a commitment to investing in new tooling, balancing infrastructure development with feature development, and a common belief that this is fundamentally a compelling shift with technical advances being considered in equal measure to feature delivery.

 

Article written by Paula Cope, Director of Deutsche Bank