Employee Spotlight: Lukas Lozovski, Director of DevOps

On a regular basis, we talk with our team members about their experience working at Canoe, and their future goals and aspirations. This week, we spoke with Lukas Lozovski, Director of DevOps.

Tell us about your background. What drew you to DevOps?

I started my career as a backend software engineer just as the DevOps movement was gaining steam. I was used to doing traditional SysAdmin type tasks with Linux servers so I was always the backend engineer that configured servers, or talked with the SysAdmins to get the servers configured.

As the DevOps movement got more popular, I realized that it was a perfect fit for me – I was always doing the “ops” portion by using my skills as a developer to automate everything. I realized that I really cared about the “meta” of software development: the software lifecycle, how the software gets deployed to the server, how a developer builds features for the software, how do we know the state of the software and the server, how do we access logs, etc. DevOps is about mediating the complex interaction between people and computers, giving people psychological safety that running commands will not destroy customer data, and empowering coworkers with the skills to build automation for themselves. 

As Director of DevOps, you’re responsible for designing and implementing Canoe’s software infrastructure. What are some of the initial steps you are taking to do this?

Canoe has seen incredible growth in the past few years. Therefore, we are investing heavily in our infrastructure so that our platform can continue to serve the growing demand of our clients. We use AWS for hosting our infrastructure and are defining infrastructure as code with industry-leading tools such as Terraform. With infrastructure automation, we can spin up and down any of our components and we can also scale horizontally. Our investment in infrastructure automation also reduces day-to-day operational support and leads to an increase in development velocity for the broader engineering team.

From a DevOps perspective, what do you see as the most impactful initiatives for Canoe right now?

The biggest initiative we have is the effort to convert everything into Infrastructure as Code. Our biggest growing pains all come from the parts of our infrastructure that are not automated. As the DevOps team writes Terraform, we’re also making sure to make our infrastructure support highly available multi-region deployments to AWS. Currently setting up multi-region application availability is blocked by lack of automation.

What does your typical day look like? How do you interact with other members of the Canoe team?

My day-to-day is quite different from the other DevOps Engineers, deliberately so. I try to keep myself out of the “hot path” for any project and instead assist every DevOps Engineer with their tasks if necessary. I attend daily standup meetings for multiple teams in addition to the DevOps team and coordinate which tasks are being worked on to balance long-term project work (Infrastructure as Code, CI/CD pipelines) and short-term support tasks (help in troubleshooting, access to resources).

One of the biggest challenges for DevOps teams, in my experience, is context switching. It’s hard to work on a big project when short-term support tasks constantly pull you away, so I try to take on the short-term support tasks to shield the rest of the DevOps team. This way I get really good exposure to various support tasks to see what we can automate, and other DevOps Engineers get to work on long-term projects without disruption. 

What’s the best piece of advice anyone has ever given you?

I don’t remember who gave me this advice, but it has served me really well in various DevOps roles: Don’t be afraid to “pull the string” to follow something to its source.

When dealing with software infrastructure you inevitably end up dealing with layers of abstraction and your ability to diagnose an issue will depend on your ability to mentally visualize the application and how the different components fit together.

A DevOps engineer absolutely has to have the skill to, for example, look at an AWS Route53 DNS entry, follow that to the Load Balancer, then Target Group, then EC2 instance to troubleshoot some issue. Almost always, the code running on the EC2 instance will be unfamiliar to the DevOps Engineer, but being able to navigate through every single layer of abstraction and understanding how it relates to the next layer of abstraction will help with identifying issues.