Continuous Integration and Delivery in an Agile Environment

  Agile is pretty commonplace in development environments, but it’s often difficult to add continuous integration and continuous delivery (CI/CD) into the mix. For older development teams, they might not even be aware that much of the push from a development codebase to testing and then to production can be automated. Agile was created by developers who found old school Waterfall methodologies were not efficient for modern day software lifecycle management, but now speed of deployments is a critical component in customer satisfaction and productivity. By adding CI/CD, an Agile development environment is much more streamlined, and developers can maintain rapid delivery of patches and updates during the software lifecycle that traditionally was much slower before automation was available.

Agile and CI/CD are Separate Concepts

Before diving into CI/CD implementation with Agile, it’s first important to understand that all three methodologies are separate entities and should be considered components that work well together instead of a whole unit. The reason it’s essential to understand these components individually is because they each can be configured and integrated based on the organization’s own requirements. The term “Agile” is derived from the term “agility.” It’s a methodology that defines the way software is created and maintained. It replaces the old Waterfall methodology popular in the 70s and 80s. In older Waterfall methodologies, key stakeholders defined specifications, these specifications were sent to developers, developers coded the application, and then the finished product was presented to stakeholders. The process could take months to complete, and any issues would not be found until completion months after development time had already been wasted. This issue created a bottleneck in the development lifecycle and cost thousands in developer refactoring and code changes that could have been found earlier in the process. Agile changed the old Waterfall methodology by allowing for changes during the development process. Development of an application is performed in parts with each part being a part of a sprint. After the sprint, the code changes are added to the main codebase and presented to a project manager or stakeholder. They sign off on the code, and then the next sprint begins. Agile has long been the preferred methodology for developers as it leaves room for bug fixes and changes before development is completed so that the entire process has a better chance of success when it’s ready for deployment to production. CI/CD are their own components that can work well with Agile. Traditionally, when bug fixes and updates must be added to the codebase, developers checked out the code file, made their changes, and checked in the file to add it to the entire codebase. The developer would then wait for the next deployment, which can be anything from the next day to a month later. If code changes created additional bugs or did not work, it went back to the developer. This process would drag out a simple bug fix by potentially months. With CI/CD, automation is added to the process to avoid costly bugs from simple patches or large updates to the code. Automation with CI/CD speeds up the process and can significantly reduce development time. Developers that work with continuous integration still check out code, but automation is added to the process where tools are used to build and test developer changes as soon as they check in code and notify them immediately if errors are found. Continuous delivery (CD) automates the process of deployment by using orchestration tools to send code updates to the target production host. Instead of waiting for an official deployment day, which is usually done monthly in large environments, CD automates the process and deploys new code as soon as it’s tested and ready for deployment. Using Agile with CI/CD, developers can more rapidly deploy updates to their software and reduce the number of critical errors in design and specifications when presenting new changes to stakeholders. The combination of methodologies defines the way modern software is developed and distributed. It removes some of the human element of software management and deployment to reduce time and the potential for errors.

Combining Agile with CI/CD

Most development teams are already using Agile, but most teams customize Agile to fit in with their own requirements and interests. Even with a custom Agile approach, CI/CD can be easily integrated into current methodologies. The key with CI/CD is with the right automation tools and procedures. CI/CD is heavily focused on automation, so developers must determine how to orchestrate changes and find the tools that work with the environment. Done right, Agile and CI/CD can save money on developer time, manual integrations, and resources used to manage them. CI/CD adds a reduction in downtime to the equation. Overall, the three methodologies can save thousands in development costs. Agile can speed up development by up to 50% reduction in time to production. CI/CD is much newer than Agile, but it’s already proved to be a benefit to both large and small organizations. For example, Ticketmaster added continuous integration into their development pipeline using Jenkins and reduced their code-to-test process from two hours to eight minutes. In addition, they were able to more rapidly deploy updates to their mobile applications. Continuous delivery (CD) also has its benefits. CI/CD is used as a part of DevOps to automate testing, deployment and delivery of applications. Research in 2019 conducted by Puppet showed that CD:

• Sped up delivery of bug fixes 24 times faster.
• Reduced change failure rate by a third.
• Developer time on rework and fixes was reduced 22%.
• Developers spent 50% less time on fixing security issues.

Before you decide to add CI/CD into your Agile methodology, there are a few aspects of software management and its lifecycle that cannot be eliminated:

• Support personnel and IT infrastructure staff that will be needed to support the software and automation tools.
• Testing — both manual and automated.
• Agile sprint meetings and the design phase of development.
• Project managers liaisons to communicate project changes and decisions with stakeholders and developers.
• Backlog management
• Security penetration testing, reviews, and remediation of vulnerabilities.

The way CI/CD is integrated into your current Agile process depends on the requirements, environment, and where the application code will be pushed to production. You can write your own automation scripts, but a much more efficient way to deal with automation is to use the right tools. Some popular tools include:

• Jenkins
• Bamboo
• GitLab
• Gocd
• CodeShip
• Travis CI

After you’ve determined the tool that you want, it should be configured to integrate with your change control solution. For instance, most tools integrate easily with GitLab and GitHub where developers check in their code on a local GitLab repository and automation tools deploy new code to GitHub. The automation tool takes code from the GitLab repository, builds it and sends it to a target server. This target server could be a quality assurance (QA) server where human reviewers test code or additional tools run automated tests. Most design changes that affect frontend user experiences are human reviewed for QA, and then tested code is approved for production. Automation CD tools promote code to production when it’s tested and approved. Now take this one code update example and multiply it times the number of developers and the number of codebase changes that they make daily. Instead of managing every codebase change including bug fixes returned during QA reviews, automation tools eliminate the need for human intervention speeding up the process and detecting errors before they go to production. During an Agile sprint, the developer does not need to interrupt a current sprint to return to previous tasks with bug issues. Instead, automation tools build the source code, detect errors, and send an email to the developer and team lead.

Following Agile and CI and CD Best Practices

Once you commit to using continuous integration and continuous delivery, it takes some upfront costs and man hours to implement it. For Agile and CI/CD to work seamlessly, developers and every person in the pipeline must adopt a mindset that code changes. Some best practices to follow to help teams move forward with new methodologies:

• Automate tests throughout the day and send notifications to developers and leads when new code fails.
• Have a staging environment that is the exact (or as close as possible) mirror of production where code is tested.
• Consider adding an automated penetration testing solution during the test phase to identify vulnerabilities before they go to production.
• Log extensively in both testing and production to help developers identify bugs.
• Continue to review and make changes to automation configurations and actions to improve the process.
• Measure your improvements including the number of bug fixes, the time to deploy to production, and the number of releases made after automation tools were put into place.