Over the past few years Mind Candy has gone through a DevOps transformation. We did this because we knew that we had to improve the delivery of our products and we knew that where we wanted to be involved having the following three things in place.
1. Shared goals and practices by aligning our different teams.
2. Unified tool sets, again we needed to align around a common set of tools.
3. Collaborative learning – knowledge sharing was and remains vitally important to us.
Obviously, achieving something like this cannot happen overnight. It had to be an iterative process just as software development is, and its starting point required changing the mindset of people across the teams so that we began to do DevOps.
These are some of the practical things we did on that journey.
Familiarity doesn’t breed contempt
In Aesop’s fable of the Fox and the Lion, we’re taught the moral that familiarity breeds contempt. However, in an organisation trying to transform towards a DevOps way of thinking we turned the fable on its head, acknowledging that it’s not familiarity that breeds contempt but separation does in the form of silos.
For us this didn’t mean that we needed everyone to know or be familiar with everything about everything. Unicorns don’t exist. What it meant was making our physical working proximity closer. It’s pretty amazing how, when different teams can hear each other – from Dev through to QA and onto Ops – how much more readily they interact and collaborate organically.
We found that technical decision-making became a much more shared process. Closer working environments encourage greater mutual support between teams.
It’s good to talk
Email is a wonderful thing. Instant messaging and relay chats are even better once you’re in a good DevOps place. However, if you’re trying to shift attitude and thinking email is not a substitute for getting up and talking to someone or having a phone/video call.
It might not always be possible across timezones, but it doesn’t take a genius to realise that intonation can easily be lost in the written word even if someone uses a emoticon.
The slowest and most problematic IT organisations I’ve known have tended to be ones where everyone hides behind email, resulting in bubbling tensions, and often leading to escalation and wars over who can CC the most senior people in. Change is able to be effected but only based on who has the loudest shout or clout.
Meanwhile, the best and least problematic IT organisations tend to be the ones where different functional teams not only sit physically close to each other but where they also walk across the office to talk to each other instead of sending snippet of easily misinterpreted text over the Internet. Obviously when you have no choice you have to use electronic communications, but when you don’t need to you probably shouldn’t.
Investment in knowledge pays the best interest
When you look up a typical DevOps venn diagram online, it will be one where DevOps sits as the joined intersection of Dev, QA and Ops. Acknowledging this intersection is crucial in moving an organisation’s mindset. The intersection represents all the things that you do that have a shared interest and investment in them. This is the place that you need to align across teams.
Take code deployment as the classic example.
During any software cycle, each team will deploy to different environments and it’s highly likely that there may be differences in the process due to the scale of environments, whether they operate under SLA, or under any internal governance controls like change management.
The tools used to deploy, and the process followed are an excellent starting point in any DevOps transformation. They not only encourage collaboration between teams, but also enable you to unify your toolset under known standards, something we have done at Mind Candy that I blogged about previously.
This has empowered tech teams to collaborate on a shared interest and shared investment, whilst also carrying a shared responsibility for its maintenance. The tool is as much a “product” as the product that it ships.
The net result of this investment is that code deployment becomes so trivial that it widens the scope of who can “push to live” to pretty much anyone. This shouldn’t be mistaken for anyone should (or does) deploy to live. That would be silly. Rather it should be seen in the terms that a robust deployment process can eliminate the lone rock star engineer being a single point of failure.
As Mazz Mosley said at Monki Gras 2013 when talking about how GDS built gov.uk, “rockstars are not webscale”.
This approach doesn’t negate strict change control and governance in the organisation (if you have it). It simply removes blockers from your delivery pipeline. Thats a win for the business as much as it is a win for those who have shared and gained knowledge through collaboration.
Devs as Ops and Ops as Devs
Once we had shared ownership and responsibility of tooling like deployment spanning across teams in the organisation it was clear that the reality of the DevOps intersection is one where Devs are Ops and Ops are Devs
This doesn’t mean that either team does the others job. This is not the full stack unicorn. Sysadmins are not dead and nor are developers, It just means that where the things they do have alignment they can learn from each other.
Take the traditional sysadmin position. They will often be quick to tell you that they’re not a developer. They may even say it with a sense of disgust that you even dared to ask the question. The sad truth is that they’re actually in denial.
They might not like it, but when writing short scripts, or declaring something in a configuration management system, they are developing, and, as the saying goes they’re doing “infrastructure as code”.
The only difference really is that frequently they have made life hard for themselves by lovingly hacking systems and creating the snowflake server. It’s great for job security of course, but it’s terrible for the business – rock star ninja single points of failure again.
At the very least they need to be using some sort of version control for the infrastructure, and what is version control if not a development tool? However, it’s not just in the tools that your Ops can be more like Devs. There’s the working practices too.
The Ops team had already been using Kanban to prioritise work weekly. Whilst this worked to a degree the team still had an ever growing backlog of tickets and requests, and what went on the Kanban board each week still contained a considerable amount of reactive work.
We decided, as a team, that we would take our workflow a step further and apply more development principles to the management of our ticket queue. We decided we should align ourselves with our colleagues and move towards a greater form of Agile along scrum lines. We would start using sprints, planning, backlog pruning and prioritisation.
We began to work through our backlog by opting for two week sprints. We introduced sprint planning, and started to commit to a certain number of story points (issues) for the sprint, and, barring any major issues or emergencies (which we left slack for) we would stick to the committed work and do nothing else.
The impact of what was a pretty small change was huge. It took a few sprints, but, as our different product teams (who were all also doing sprints obviously), became aware that we working in the same way as them, emergency work and high priority issues out of the blue gradually declined.
Obviously it’s not always like that when you’re supporting live services as well, but, by aligning our working practices with our primary internal customers, there became a greater appreciation of how our backlog could be impacted just as theirs could be by altering the scope of the sprint.
This was indirect collaboration born on the back of working in a more aligned way with our peers. Our backlog went from over 100 tickets to less than 40.
Meanwhile, as we in Ops were being more like Devs, we started to share some of our Ops roles with Devs with a little help from our a friend called Canbot.
ChatOps sets you free
Candy Bot, or Canbot for short is our in-house name for Github’s Hubot. It sits in our dedicated Slack channel #chatops and when not providing us with amusing animated cat images he/she does things for the Devs and for Ops.
Canbot can tell us where servers are. This is vital as we use AWS so the environment can be fluid and dynamic. Canbot can deploy config changes for the Devs to each environment, including to live and it’s all totally transparent.
If someone changes the code base in our Puppet infrastructure then Canbot will tell #chatops about the commit and who did it. We also opened up the Puppet repository to the Devs and some of them change it every now and then. Shared responsibility after all.
Canbot can also execute commands on our infrastructure, but when it does it is never in secret. Transparency is the key feature here. What Canbot can do is also open across the teams for development. Primarily it is Ops that play with him, but there is nothing stopping a pull request from others internally.
Canbot has allowed our Devs to be a bit more like Ops. They can orchestrate production without having to have ssh access and it can be audited. No more tickets asking for information about production.
Embrace failure
Failure is an opportunity to learn, it is not an opportunity to point a finger of blame and start shouting at someone. DevOps mindsets should see each failure in these terms. Iterate the failure and eliminate it with either better toolings, better documentation or better gated processes.
When we celebrate failure we do it with KrispyKreme donuts!
Encourage Tech Culture
Most of the people that work in tech love tech. Few of us see our jobs as a mere means to an end. If you encourage your technical teams to collaborate with learning sessions too you can create a greater sense of being “one team of many disciplines” rather than single teams doing only one thing.
At Mind Candy we hold regularly weekly book clubs open to whoever wishes to join, where we go through a particular book on a technology matter. We also have Guilds where we present and share what we’re working on between teams.
Additionally we use our office as a host location for MeetUps across tech businesses. Next month we’re hosting a London Virtual Reality meetup. Sharing should not always just be in-house after all.
Wrapping things up
Obviously the list and experiences above are not exhaustive. There are so many little things that an organisation can do when adopting a DevOps approach. What’s important is to realise that you change the mindsets first and then you iterate and encourage greater collaboration. Once an IT organisation realises that it relies on mutual support to sustain itself change can come about quite rapidly.
