When it started to really get traction as a concept, almost 10 years ago, DevOps was primarily used to push rapid changes to web environments with minimal impact on the users. That might still be true today but DevOps as a concept evolved. A lot.
We now rely on DevOps models to move at high velocity, adapting and developing at speeds that are light years away from anything we’ve seen before. It’s the way we deliver, test, monitor, and release functionalities. A strong DevOps culture will help teams collaborate better, reduce back and forward, and develop new features without sacrificing security along the way.
On the other hand, however nice that may sound, making the change to a DevOps approach is not that easy. Besides the proper processes, more than anything, you need the proper team, which we are going to discuss today.
We reached out to a few awesome companies and poked our noses in their day-to-day operations hoping to find out how they managed to get over this major hurdle. We asked how they manage to build a successful DevOps team, what are the main roles you should have to get started, and how you can make sure they work seamlessly together in responding to DevOps challenges. We also poked our noses in their tool choices hoping to find a balance between building vs. buying new tools. You can read all about it in my other article about how to create a healthy DevOps toolchain.
DevOps Team Structure: What Are the Roles and Responsibilities of a DevOps Engineer
So what exactly is the role of a DevOps Engineer? There’s no easy answer here as most teams have very different requirements but if I had to sum up a few characteristics that define a good DevOps engineer it would be something along these lines: A person that can act fast and be agile enough to shift focus at a moment’s notice. This person needs to be comfortable wearing more than just one hand as most often than not, he or she will have to take on uncomfortable or unfamiliar tasks in order to move forward.
Najib Radzuan, a DevOps lead from Malaysia has a very simple yet quite clear perspective when it comes to the DevOps roles within an organization. Here are the five primary roles that should be in your team structure:
DevOps Evangelist
The person must proactively create a good rapport with all teams involved in the software development and IT Operations team. His responsibilities include strategizing and planning for DevOps adoption within the organization as well as finding the best platforms to increase productivity. If you’re an Evangelist and want to optimize your technology stack, check out our blog post about the top DevOps tools available today.
Release Manager
Would be the person in charge of every new release and would have to oversee the coordination, integration, and flow of development as well as testing and supporting the CI pipeline. They wouldn’t only create but also make sure the application delivery toolchain is thriving and functions at peak performance.
Automation Architect
The automation architect would have to identify the different automation opportunities within the development process and the testing process. He would design the scripts that the team would be using while developing, testing, and monitoring the application.
Software Developer
Developer consists of front-end and back-end people who are actively doing development. They work side by side with QA to make sure the code is delivered bug-free. The developers are responsible not only for turning new requirements into code, but unit testing, deployment, and ongoing monitoring as well
Security Engineer
For the better security and compliance of our apps/environments we need a person that oversees this area. This role works closely with the IT Ops team to plan the best approach for the apps/services. The Security engineer must work with both internal and external teams to ensure apps/systems are securely integrated, configured, managed, and supported in production.
How Do You Create a Highly Effective DevOps Team
Now that we’ve looked at some of the most important roles you need in your team structure, we are undoubtedly left with the question of how to create a successful, effective, and collaborative DevOps team. Here’s the recipe that we’ve found:
Attracting and Retaining the Right Talent
I can tell you right away it’s not going to be easy. Finding a team of individuals with the talents to take on tasks at both ends of the DevOps continuum, companies are going to have to make these roles lucrative if they wish to attract talent. I realize this sounds like a disappointing outcome, that not every company can reap the benefits of DevOps overnight. I think that DevOps as a whole is still in its infancy and we need to let it grow as naturally as possible over the next few years in order to better understand what a more long-term vision looks like. ImagineX software consultant Scott Simontis thinks so too:
Long-term change is going to require bold action from employers who desire to attract the talent required to organize their own technical “strike force” teams who have the skills and industry experience required to balance Development and Operations without requiring too much external support.
This fundamentally changes the team dynamics in a way that previously happened by coincidence, if it happened at all. Instead of having highly specialized team members, you need well-rounded and experienced generalists. Ideally, someone on the team should be able to write a front-end component, create the API that the component needs to call, design the database schema to support that API, and then be able to get the whole solution running given nothing but some virtual machines or a cloud instance. This approach makes it impossible for there to be a wall between Developers and Operations, because “DevOps” is now part of the definition of complete code.
While having really great developers specialized in individual programming frameworks and languages is great, a highly effective team will have people that can jump from back-end to front-end and be more of a jack of all trades rather than a one-trick pony.
Use Smaller Team Structures
Smaller teams seem to be what high-performing DevOps organizations favor since they make it easier to adopt the loosely coupled, microservices style of software delivery. As Scott puts it:
This structure isn’t possible everywhere and finding individuals interested in acquiring such a broad set of skills is rare. However, when it can be realized, I believe it results in the most effective team. With this approach, you would have a number of small-sized teams (3-4 people) all working together on the same domain object.
We must change our view of software and adhere to Domain Driven Design if we wish to divide an application into a reasonable number of teams that can operate in parallel, and it sets the project up for a possible microservices implementation right from the beginning, should a microservices approach make sense for the problem at hand (often it doesn’t). You’ll still see Developers and Operations outside of the team working on projects that don’t lend themselves to this domain focus, and those staff members can still serve as mentors and offer Subject Matter Expertise when the team reaches their limits.
Provide Proper Tools and Training
Both DevOps newbies and oldies like to work with shiny new technologies and methodologies. You’ll need to create an innovative and healthy toolchain that keeps them excited, helps them be more effective and, often, also play a role in recruiting and retaining talent. As Pavan Belagatti, Growth Marketer & DevOps Advocate at JFrog, put it:
That is the basic concept of all these modern methodologies. Lean, agile, and DevOps, all come with a vision of breaking the old methods and norms. A growth and unified mindset is all you need to break the silos and achieve things. Starting a DevOps culture is one part, and the other part is to provide training, tools, and all the necessities needed to break the old habits. With a strong desire, good hiring, skills, training, and practice, traditional teams can break the old attitudes and can transform themselves towards digital transformation.
Breaking the routine of going to the same office as the rest of your team can be tricky and requires a strong distributed team, the right tools, and lots of training. But it can most definitely work.
However, choosing the proper tooling is no easy feat. There’s a wide selection of tools available and you need to do a lot of research beforehand. There’s also the option of building your own. Regardless, both approaches have their ups and downs. I discussed this in my other article about creating a healthy DevOps toolchain.
Speaking of proper tooling, our friends over at Serialized.io wrote a great article on “How to add logging to your Dropwizard application with Sematext”. It’s a great read so make sure you don’t miss it.
Automate and Streamline Tasks and Workflows
While some companies had years to ease into a distributed workforce, a lot of companies did not have that luxury. The idea to develop new methods of interacting with your colleagues can be a bit daunting and some even go as far as to say the traditional way of developing these communication channels isn’t really all that effective. Richard Lenkovits, a DevOps Specialist & Full Stack Developer thinks that the way to a fully functioning DevOps team is not by creating more processes they have to follow but to streamline the ones they already have.
The solution here is always to define good development practices. And I’m not talking about readme files and company wiki pages here and there that you have to maintain and keep up to date, thus creating an n+1 piece of bureaucratic overhead task. Once you forget about it, nobody reads it and the whole process is a mess. I’m talking about automated checks on git commits. Static analyzers, linters, automated checks, and tests, that push people to comply with processes. Don’t be the developer’s kindergarten teacher. Write her in python.
Encourage Team Collaboration and Communication
These processes that Richard is talking about are one of the key components of the DevOps methodology. Some might argue that the processes themselves are critical but Marcello Marrocos a consultant and DevOps Advocate seems to think that people > processes.
The three pillars of DevOps are People, Process, and Products. When we talk about bringing teams to work together, that’s on the People pillar.
It is the cultural transformation required to focus on collaboration and communication, increasing the autonomy and blamelessness of members with the full support of leadership, focusing on the continuous improvement of the process, supported by the products.
At the end of the day, it comes down to the people. Not the tools. Not the processes. Your colleagues need to adapt to the new situation and find ways to communicate and get an easy way to provide updates and discuss progress.
We spoke to Andrei Pirjol, a DevOps engineer and he had a simple solution to this new order of things.
Regular standup meetings (performed in an agile environment) can help other team members, other teams what a person is working on and permit more insights to offer other people that they may want to know.
A model that they are not familiar with can have a dry run(test run) approach, select a few people from each team and work in the model presented to them to see the benefits and negative parts of it.
Cultivate Leadership and Autonomy
On the same note, Najib Radzuan, DevOps Lead, stresses the subject of anatomy vs. control and finding a middle way to assure team members have a sense of autonomy and control.
Many research shows that when people have little sense of autonomy and control in their work, there is more stress and more burnout. One way DevOps leaders can help fight burnout is to create more autonomy in their teams and not to impose restrictions on them. This means that leaders should not make all the decisions that affect team members, but rather allow them to make their own decisions.
Scott’s on the same page:
This will involve giving them more autonomy than I imagine a lot of companies would feel comfortable with allowing. Trust will be crucial to letting these teams organize themselves and learn what is effective and what needs more effective implementation next time, but that’s the price of doing business.
Measure the Effectiveness of Your Team Structure
Naturally, once you get your DevOps team going you’ll want to track their effectiveness and the best way of doing it is by looking at KPIs, key performance indicators. These can give you ideas on how to make processes run smoother and remove friction from within the team.
There are lots of different indicators you could be looking at but the three main ones are as follows:
Deployment frequency
This refers to the number of deployments your team will be doing each day. I’d suggest looking at this particular number often and making sure it aligns with the goal of your company.
At Sematext we’ve built this nice Github integration that uses Github Webhooks to push data about commits, pull requests, branches, merges, etc. into Sematext, which then lets us understand what’s going on with development across all reports and all people, down to individual repositories and even individual developers. Here is an example Team Pulse dashboard:
If your team uses Github, you can learn more about this Github integration to see how to set this up for your team.
Volume of introduced errors
Every new iteration of the code or every new feature pushed runs the risk of introducing bugs and incompatibilities issues. Measuring the number of these types of issues introduced with every push can help you understand the effectiveness of your team.
Security
Your application is a living breathing entity that grows and scales based on the requirements of your users. Frameworks will be updated and plugins will be swapped. New features will be created and all of that new code can introduce a number of security vulnerabilities that can severely increase the surface of attacks from a mal-intended person. This too is an important metric that’s worth keeping an eye on.
What Happens Now?
To anyone thinking otherwise, DevOps is here to stay. It’s a model adopted by every big company out there, that seeks to move fast and be agile, and focuses on security by following DevSecOps practices. DevOps teams have evolved over time and while I’m sure they will continue to do so in the coming years, I’m fairly certain we’ll see more developers leaning towards ops and vice versa. We’ll get to see more people that can wear multiple wigs in the team while the so-called, one-trick ponies will be slowly phased out.
To have a fully functioning DevOps team structure there are three things that need to change. The team needs qualified leadership to help them through the process. They need to focus on creating proper processes that help the team keep track of the progress without adding more bureaucracy to their day-to-day lives. The next thing they need is proper tooling. Working in modern distributed teams will already add to their already difficult job so having the tools they need to monitor and debug their infrastructure and application is going to be a crucial aspect.
I want to thank all of the people that agreed to talk to us about the challenges of moving to a distributed team and want to launch an open invitation to any DevOps engineers that would like to talk about the challenges they had to overcome.