Real CI/CD Is No-Touch

When it comes to continuous integration/continuous deployment (CI/CD), the idea often does not match up with reality for many organizations. This was pointed out in a tweet that went viral from 2020 which stated, “Let’s stop fooling ourselves. What we call CI/CD is actually only CI.” What does this mean and why did it cause so much of a stir in the developer community?

There is a distinct difference in the way CI/CD is supposed to work in theory and how it actually works in practice for many organizations. Real CI/CD is no-touch – that means that, once the pipelines have been set up, there should be no need for any kind of manual intervention in the entire development, testing and deployment pipelines. To borrow a phrase from Apple, CI/CD ‘just works’. Except, in practice, this is very often not the case for numerous reasons that will be outlined below.

Many teams have separate processes/pipelines for Dev, Sec, Test (QA), and Production 

A proper CI/CD pipeline will touch on every area of software development due to the nature of the methodology. In CI/CD when a developer commits code to a repository, it should automatically have test suites run against it, perform security scans and then have the deployment pushed to production should it pass all steps of the process according to policy. This process touches on areas that have traditionally been the responsibility of separate teams or departments, such as developers, QA, Sec and IT operations. 

True CI/CD requires these teams to work together, rather than separately. This need is partially responsible for the rise of DevOps – the integration of developers, IT operations and testers all under one roof in order to unify teams and development work. In order to best utilize the benefits of CI/CD, it must be implemented in a way that keeps all departments together. This is because this process touches on so many different areas and will only become more efficient if they stay coordinated with one another throughout its execution. True CI/CD is not gatekept by teams but by policy – should code pass all requirements, it should then be merged and deployed to production without waiting on another team’s sign-off. This means working together in true CI/CD is crucial in order to help catch bugs before they reach production.

Handoffs are time-sensitive, complex and opportunities for failure 

Organizations stuck partway between their own implementation and true CI/CD very often make the claim that they can’t do so for a variety of different reasons, from technical to cultural. As with most improvement opportunities, resistance to change because ‘this is the way we’ve always done it’ can be overcome. The short term benefits of making the change include speed, quality and efficiency; yet long term it’s about pulling ahead of the competition or being left behind. 

Traditionally, software development has been split across separate developer, tester and IT operations teams and requires hand-offs at various points in time. Each team has its own set of procedures and responsibilities and when each team’s phase in the software development life cycle is complete, to borrow a proverb, the code is ‘thrown over the wall’ to the next team. Except, this results in much slower learning cycles, miscommunication and a lack of ownership, which in turn leads to opportunities for failure. When teams are disconnected like this, the handoff becomes a bottleneck. Ideas, intentions, and justifications for why code is the way it is are lost, jumbled and misunderstood. 

Don’t underestimate how much of an issue miscommunication can be, either. To borrow another term, there is the concept of ‘technical debt’ that must always be paid at some point down the line. Technical debt is used to describe quick fixes and hacks that solve problems in the short term but cause problems in the long term due to their hacky nature. When teams are disconnected, problems that arise are often fixed with the equivalent of programming duct tape, instead of  a proper solution when different teams work together. As more and more technical debt accumulates, innovation suffers as development teams spend their time on fixes to keep the product working.

Single self-service pipeline is the future 

Self-service pipelines break down the barriers that are put up when teams work separately. Without a self-service pipeline, a developer may need to wait days or even weeks from their code being pushed to the main repo before it is actually pushed live. When teams work around the clock from around the world, this linear process soon falls apart. 

When it comes to larger organizations, developers may be introducing code changes into repositories at any time of day or night from anywhere in the world. Self-service pipelines help to make sure that QA works with developers  to ensure test suites are fast and reliable while operations staff also collaborate to ensure that the infrastructure is in place to allow committed code to automatically have changes pushed to production if it meets policy requirements. In this way, it doesn’t matter when or from where a developer pushes new code. Day or night, working remotely or locally, changes can be pushed and deployed without any technical or bureaucratic barriers, because the process and compliance requirements are enforced by the pipeline itself.

Manual approval gates only as required (ex. PCI compliance requires a separation of duties, developers can’t push to production) 

Manual approval may still be necessary in some instances, such as software that deals with life or death or in order to comply with regulatory requirements such as PCI,  which requires a separation of duties so developers cannot push directly to production without approval. These scenarios should not prevent teams from making  the transition because it is perceived as difficult or unfamiliar, as there are many benefits that can still be realized, without creating an overly burdensome process.

Generally speaking, code and technical challenges should not prevent teams from implementing no-touch CI/CD. Outside, non-technical requirements placed on a project may require some steps that keep the process in policy, such as an approval gate prior to production deployment. Nearly all the benefits of automation and collaboration true CI/CD deliver are still achievable for these  software projects as well.

Breaking out of inertia

When it comes to making changes, it’s human nature to prefer what you are comfortable and familiar with over something new and unknown. “Better the devil you know than the devil you don’t,” as the saying goes. As this applies to software development, this means teams very often prefer to stick with the procedures and methodologies they are comfortable with, even if they know they are inefficient. This is more of a psychological and organizational challenge than a technical one, but it does warrant consideration.

However, teams reluctant to change will, sooner or later, find themselves left behind the competition. Software development is a highly competitive market that evolves very quickly and requires constant vigilance to stay on top of the game. There are good reasons for this, too. Teams no longer directly upload code changes over FTP to production anymore, for example. 

Change is easier to understand and integrate if teams understand the impetus behind the changes. We no longer upload directly to FTP because it is insecure, difficult to roll back, and prone to causing website or app failure. IT downtime is increasingly expensive – one report estimated the average cost downtime to be $5,600 per minute.

Many of these new changes and methodologies become popular because they provide additional security, stability or assurances to help maintain stable, working production code. When downtime and mistakes are costly, teams cannot afford to make errors. Teams also cannot afford to act slowly or indecisively and stay competitive, they need to move fast without breaking things. This is where the true benefits of real CI/CD shine. Properly implemented, CI/CD enables developers to push code at any time that is automatically deployed within minutes of being committed. This enables fast development speed without sacrificing security and quality, something desired by many software teams looking for a way to stay both secure and competitive.

Learn more about the Guide Rails software

Getting the infrastructure set up and implemented in order to enable true CI/CD is not easy, even for teams with existing CI/CD pipelines today. If this is true for your team then get in touch with us at Guide-Rails_®. 

Our team of experienced developers and engineers can help guide you through every step of the process, from transitioning your old methodology to getting the tools and services in place to get your current projects up and running using true no-touch CI/CD. The hardest part is getting started, contact us and we’ll help get the ball rolling.