TL;DR: when are we going to get all these Open Source cloud projects in a room together with developers from both sides of the stack so we can understand how to work together better?
Recently in Berlin I attended the Kubernetes Conference which is the latest coding framework to be released by Google. Kubernetes enables Application Developers to deploy their applications to their users in what is known as ‘continuous integration continuous development’ <– note the recursive iteration of the acronym: CICDCICD, etc.
The purpose of CICD frameworks like Kubernetes (aka #K8S) is to enable the next generation of cloud application development teams to have a toolbox of “CICD pipelines“. These pipelines are ways in which to deploy your application on the cloud to your users via various audience profile segments. In short, CICD is a social engagement methodology which will empower your product team to continually innovate without making too many of your users upset as you continuously upgrade your digital product.
For example, you might want to role out a new feature button for your app, however you only want to roll it out to your power users (aka the top 10% of your customers who use your tool). This is known as a “canary deployment pipeline”. You can imagine the other kinds of CICD pipelines an application team might want to use:
- regional deployment pipelines to test a specific language or geo specific feature, e.g. a new tax widget for your employees who use your HR system in Australia and New Zealand.
- skill level pipeline roll-outs, i.e. you might want first time users to only have access to a minimum viable feature set of your app until they are ready to have additional features/buttons made available to them as they increase their skill. FYI there was a fascinating debate in the conference hallways discussing if using age data on your users was an ethical way to approach a CICD pipeline roll-out.
- closed beta pipeline rollouts, such as a new mobile app to compliment a website, with automatic in situ failure rollback. Also know as blue/green CICD pipelines.
- NB there are also a ton of pipelines all about testing your application prior to integrating with your customers.
Yes, this new toolbox of CICD pipelines is a methodology, and likely to be as popular as agile/scrum was this previous decade. However, it still has a glaring problem which needs to be discussed. This problem was nicely outlined in Berlin by Michelle Noorali from Deis (which is a startup product that integrates CICD pipelines using K8S, recently acquired by Microsoft Azure):
The problem which Michelle outlines via her “broom/vacuum/roomba” metaphor is that application developer teams are still using vacuums and are still trying to agree which kind of robot vacuum-cleaner they should jointly use?
Forrester’s report on “Container / Platform as a Service” products earlier this year, supported Michelle’s point highlighting that there are over 70 projects which are integrating CICD pipelines into their product toolbox. The majority of these products are still very beta, without evidence for which pipelines will work best for different application developers teams? Which of these C/PaaS are going to emerge as the right set of CICD pipelines suited for varying application development teams?
To name a few of the dominant C/PaaS who are enabling CICD-like features, including K8S capabilities: Cloud Foundry, RedHat’s OpenShift, Apache’s DC/OS, Cloudify, Microsoft Azure’s Deis, Google Cloud Engine/Borg (FYI Borg is the original code base which Google Open Sourced as K8S).
The core problem here is methodological. Getting an application developer team to agree which framework they should all use to develop their product is the proverbial herding of kittens (Netflick’s recently claimed they have 9,000 CICD pipelines, and want to agree some shared templates amongst their 4,000 employees):
To compound the problem, you also need to engage developers further down the stack to get agreement with your application developer team (i.e. remember all these folk who get your application to work worldwide: SROs, DevOps, SysAdmin, NetEng, et al). These ‘lower down the stack’ cats are much less likely to embrace change rapidly.
What is the answer for finding out which pipelines are the right ones for your team to use?
My answer: Open Source CICD Pipeline Hackathons!
- build all these Open Source CICD enabled C/PaaS onto the clouds.
- invite/train application development teams how to use these pipelines for the competition.
- bring these teams into the same room as the DevOps/SysAdmin/NetEng who administer the C/PaaS and can provide mentorship to the competing teams.
- have a great weekend hacking and learning with one another in a fun yet informative competitive training environment.
- Survey/interview the teams (behind closed doors) to provide feedback to the C/PaaS re the advantages/disadvantages in using the pipelines.
- Promote the winning hackathon teams to the world and let them speak to the advantages of using CICD.
- Rinse, wash and repeat!
We are now at the edge of computing where engineering science meets social science. CICD pipelines are where your customer team needs to agree the ‘digital playing field’ where your product will be improved together with your infrastructure engineers.