Hiring DevOps Engineers

I have interviewed many DevOps engineers in my time and I've hired some amazing people as a result. To understand what makes a great DevOps Engineer, first, a definition via Gemini:

A DevOps engineer bridges the gap between software development and IT operations. They automate and streamline the software delivery process, building systems that allow development teams to release code quickly and reliably while maintaining robust infrastructure stability.

To break this definition down into the pieces that make up a list of interview questions I might ask, let's get a bit more granular about some of these.

bridges the gap between software development and IT operations

To me, this means an engineer with experience understanding the Software Development Lifecycle (SDLC), the practice of developing, testing and shipping code and understanding important facets of that including security, logging/feedback, observability and sustainability (the bridge part!).

automate and streamline the software delivery process

This would include developing pipelines in continuous-delivery tools, adding GRC (governance, risk and compliance) related gates into those pipelines, developing clean observability with alerting-and-monitoring, and automating all of this with a process like GitOps.

maintaining robust infrastructure stability

This might mean developing infrastructure in the public-cloud with IaC (Infrastructure-as-Code) tools like Terraform, CloudFormation or other APIs, which would also include managing the lifecycle of the infrastructure that hosts the code we deploy.

What's interesting about some of the engineers I meet is that each of these varying skills are sometimes "siloed" at some companies, where there is segmentation-of-duties across this space. By this I mean:

• Software Engineer:
  Develops code and moves it to the proper branch for deployment, and may own some of the CI process for tagging, branching etc.
• Release Engineer:
  Owns the code-repositories, pipeline lifecycle and release tagging, merging and even the release of code
• Pipeline Developer:
  Develops the end-to-end pipelines as part of the CICD lifecycle and ensures all GRC gates are in place to ensure software quality
• Infrastructure Developer:
  Develops IaC only and maintains state of all cloud based infrastructure
• Operations/Observability Engineer:
  Owns the observability, feedback, monitoring, alerting, paging and incident processes related to code running in (usually) Production environments. This is typically spans a variety of third-party software to manage as well

This varies greatly from company-to-company, but at least for the teams I've managed, understanding how to wear each of these hats is important for the work that the DevOps engineer owns.

What's unfortunate with companies that do silo these functions, is that the operational knowledge of these engineers is focussed on a single facet of the end-to-end SDLC and DevOps process.

When interviewing engineers who've either been siloed into this situation, or have chosen to focus on one part of this process, they tend to show that they've become quite weak on the other components that make a strong DevOps engineer.

That now begs the question:

For me, build something that other people will use: a website, a simple API, a web-app or software tool. Show me:

• You've actually programmed something that works, that has tests both as unit-based or pipelined (or both).
• There exists a build process based on a tag or branch and releases adhere to these items as the versions of your software
• You host your software somewhere (public or private) and there exists an automated pipeline that runs to build, test and deploy the software with good feedback about operations (smoke-test), helpful logs, and telemetry to help fix things if they break
• You might have set up observability on your software including monitors and alerting based on logs/metrics or personas to continually test and alert if they fail while using your software
• There exists good, well architected security practices around your software like Authentication, Authorization, network policies, rate-limiting, encryption, secret management, frequent API-key rotation, input-validation and more
• Perhaps most-importantly, despite the ability for AI to do almost all of this for you, understand why these things are important and have opinions about certain ways you prefer to develop and implement these items

My interviews

I like to understand what interests you in applying to the DevOps engineering position I'm interviewing for. What motivates you, what interests you and perhaps how do you want to be challenged?

I will drill in to specific items on a resumé, specifically if you've listed you're an expert in something; truth be told, I will go deep on these ones!! Usually this is a general discussion about what you've achieved and what you might be proud of (and why!)

I will ask questions about a hypothetical application planned for public consumption. There are no wrong answers here, but be forewarned, this is where I find out if you're able to wear all the hats or not based on the breadth of your answer (and experience...).

Perhaps most important, the engineers that I have hired and seen bloom into extremely strong and relied-upon DevOps Engineers are the people who:

• are enthusiastic about problem solving,
• don't complain about on-call,
• are efficient and diligent about delivering on their tasks or projects,
• have a strong technical and operational breadth and understanding of the above mentioned areas,
• are able to understand other company-team-problems and how they might relate to items DevOps might own,
• aren't afraid to say: I don't know...
• develop strong relationships with their team and the organization in general

This may sound like I only hire Senior to Staff or better DevOps engineers. This is simply not true. I've hired engineers who at some companies are considered junior or intermediate into Senior roles. I've even hired Staff engineers who weren't super strong at programming, but could problem-solve, develop hypotheses and even stub out some pseudo-code.

Does this sound like you? Keep an eye on the job board at Auvik where I am usually hiring new DevOps engineers into my team!