In today's fast-paced software development landscape, predictability and efficiency are paramount. dotCMS has enhanced our customers' experience by embracing continuous delivery (CD) practices. This strategic move has yielded significant improvements, making software delivery more predictable and reliable.

What is Continuous Delivery?

Continuous delivery is a software development practice where code changes are automatically built, tested, and prepared for release to production. It ensures that software can be reliably released at any time, which is crucial for maintaining a competitive edge and responding to market demands promptly.

What We’ve Changed So Far

There are so many adjustments (big and small) that get made along the way. Here are the biggest ones we’ve made so far.

1. Containerization & Orchestration: We changed the way we package and run dotCMS. By leveraging containers to make consistent builds, we achieve consistency throughout environments. By adding on Kubernetes for orchestration, we can declare how we want dotCMS to run and let the machines handle the container’s lifecycle automatically.

2. Capture the Flag: Feature flags have become an integral tool for us. These flags allow us to add amazing and complex features incrementally, finding market fit as they evolve.

3. Trunk-based development & Continuous Integration: We use short-lived branches and frequently merge changes back into the main codebase (trunk). Each integration triggers an automated build and test process,ensuring that the codebase is always in a deployable state.

4. Release, Release, Release!: We focus our efforts on making releases as “effortless” as possible. As our release cadence increases, we continue to find and address new bottlenecks so that we can continue to increase the frequency and reliability of our releases.

5. Autonomous Cross-functional Teams: Moving quickly and safely requires a great deal of context, focus, and expertise. \We’ve empowered our delivery teams by making sure they have all the diverse skills you need to execute within the team, and we’ve improved the agency of the teams by focusing them on specific user personas.

Our Journey to Continuous Delivery

Before we started adopting continuous delivery, dotCMS faced several challenges common in traditional software development methodologies. Releases were infrequent and could be downright painful at times. They often took the form of projects that spanned weeks, requiring extensive manual testing, coordinating change freezes, and generally making folks nervous. This approach led to even less frequent deployments, which ultimately delayed features, created longer feedback loops, and ironically increased risk;

Recognizing the need for a more agile and responsive development process, dotCMS began implementing continuous delivery methodologies in 2022 with the end goal of a full transition to continuous delivery. This shift has had a profound impact on our entire company, from how we organize teams to how we build, test, deploy, operate, and support our product. 

While our journey is not yet complete, I thought it a good time to share some of the progress we’ve made and the results we’re seeing. So let’s take a look at some of the data.

Note: At the time of writing, we’re halfway through 2024. To help make comparisons easier, I’ve visualized the first and second half of the years in different colors.

Our first big breakthrough came in 2023. By leveraging containers and feature flags to encourage smaller and more frequent merges, we were able to grow the rate of changes from 855 in 2022 to 1429 in 2023, that’s a 67% increase! While the rate of increase has leveled off in the first half of 2024, it’s great to see that our current rate frequency is sustainable and remains at a solid 5% YoY growth trajectory. 

It quickly became clear that smaller changes were helping us get code into trunk faster, but we still found ourselves with a similar problem to what we started with. From the customer’s perspective, the changes were still big and painful. Since not many people use our code directly from trunk, no one would get to use these changes until they were bundled up and deployed as a release. So in the later stages of 2023, we begin really prioritizing releases. 

We needed LTS-level reliability for all our releases, and we needed to get LTS patches out faster to keep up to speed with our increased merge frequency. By extending the usage of containers and feature flags, and really getting serious about trunk-based development and continuous integration, we unlock our next breakthrough… 

And what a breakthrough it is. In 2022, we produced 8 releases. By the end of 2023, we had shipped 43 releases, an over 400% YOY increase! What’s really exciting is that we still see huge runway available for growth here. So far, in just the first half of 2024, we’ve shipped 70 releases, that’s trending at another YOY increase of over 200%. 

There’s another important dimension to consider here, and that’s the reliability of what we’re releasing. It does no good for us to change lots and release quickly if what we’re producing is buggy and frustrating to use. The Law of Large Numbers in probability theory gives us some guidance here. 

Paraphrasing here, the more of something you have (i.e. changes) in a group (i.e. release), the more likely that at least one change in the release will be defective. Probability multiplication, complexity, and resource strain all mix together to increase the probability of an escape defect on a release as the number of changes per release grows.

And here, by looking at the number of releases published and the average changes per release, we can see the amazing progress we’ve made in just a short period of time. Along with our wonderful increased change and release rates, we also see a strong downward trend in the riskiness of each release as governed by the Law of Large Numbers and measured by the number of changes per release.   

We’re not done yet. Next on the horizon for us are opportunities to improve how we conduct automated compatibility testing and how transparently we can deploy to live environments. 

Let’s Take a Look at Some Additional Learnings

1. Smaller is Better!: Smaller, more frequent releases start with smaller work items. Having well-refined and understood user stories is the key to keeping pull requests small. Small pull requests are easier to understand and faster to review, giving us short-lived branches and small, predictable merges to trunk. 

2. Automated Testing: Automated tests run at every stage of the delivery pipeline, ensuring that any issues are identified and addressed early. This leads to more stable releases and fewer last-minute surprises.

3. Consistent Release Schedule: Our progress has enabled dotCMS to establish a consistent release schedule. As seen in the release data, the number of releases has increased steadily over the years. For instance, in 2024, dotCMS has already achieved 70 releases, compared to 43 in 2023 and just 8 in 2022. This consistency helps stakeholders plan and prepare for updates more effectively.

4. Reduced Downtime: By automating the deployment process, dotCMS has minimized downtime during upgrades. This ensures that new features and updates are available to users with minimal disruption, enhancing the overall user experience.

5. Improved Feedback Loop: Our progress so far and our commitment to eating our own dog food enable faster feedback from end-users and internal stakeholders. This feedback is crucial for making quick adjustments and improvements, further contributing to the predictability and quality of software releases.

Conclusion

Adopting continuous delivery practices has revolutionized dotCMS's software development and delivery processes. By enabling frequent, automated, and smaller releases, dotCMS has achieved a level of speed and predictability that was previously unattainable. This transformation not only benefits the development team but also enhances the overall user experience by ensuring timely and reliable software updates.

As the software industry continues to evolve, dotCMS's commitment to continuous delivery serves as a testament to the value of embracing modern development practices. The journey to continuous delivery is ongoing, but the results so far underscore its significance in building and delivering high-quality software predictably.