Enabling and maintaining a cost conscious culture. At every level.

The advent of DevOps came about at a particular time in IT when there was a distinctive move away from team silos and rigid designations between areas of responsibility and ownership. Anyone who has lived through the 80s, 90s, and even early 2000s ITIL-flavoured IT outfits will probably have a lot of interesting anecdotes to tell about the good old days, but will undoubtedly admit to preferring the way teams collaborate nowadays to deliver high-quality software.

Since those first DevOps conferences, we have seen the emergence of a wide array of new titles and roles (SRE, DevSecOps, Cloud Engineer, Platform Engineer) as well as the broadening of responsibilities of the “traditional” teams made up of software engineers. Today, teams aren’t exclusively responsible for building and delivering narrow product features, they also monitor their application's performance and dependencies. They ensure its reliability in production and might even be part of the on-call rotation, making them grumpily fall out of bed at 4 am to respond to an outage.

By spreading the responsibility and blurring the lines between the old Dev vs. SysAdmin teams, software development is as good as it’s ever been. Metrics around software delivery speed, product iteration, bug fix response times and application reliability and many more have drastically improved.

The budget conservation alignment issue

In the post-DevOps world, teams are generally able to juggle more responsibilities and cover a larger domain space, but one responsibility that has rarely been aligned with the incentives of the tech teams is that of cost optimization.

I’ve noticed it to be a sticking point at all of the companies where I have previously worked. In the absence of any direct responsibility for cost allocation matters and resource spending, there were always extremely weak or nonexistent processes around provisioning new resources or extremely vague guidelines around the cost management of previously provisioned resources.

For instance, I recall while previously on a DevOps team working for a SaaS company where I received a ticket from a developer asking to “beef up” a particular database instance. Since my manager had recently asked my team to start being more “strict” about resource spending without giving us any guidelines on how to enforce or on what to base that “strictness” on. I asked for the developer to at least add how much they thought that increase of instance size was estimated to be, and if his team leader would sign off on the expenditure. He was taken aback by the question since nobody had ever brought up the dollar cost of a resource request in the past.

This is to be expected to a certain extent, reducing costs and keeping an eye on the bottom line is a KPI that usually resides a little bit higher on the managerial totem pole. It’s usually the job of the CEO and CFO to lean on the senior managers to tighten their belts when they see a worrying cost projection trend. But it doesn’t have to be this way. I would argue that making technical resource cost optimization part of the team culture can have positive ramifications and benefits at every step of an individual's career and therefore the team and company at large.

Frugality should not be the “be-all and end-all”

Just as a quick disclaimer, I’ve heard the horror stories coming from companies with frugality as an integral part of their business model. It can be incredibly corrosive to team morale when a company won’t even sign off on a top-performing team making some personalized mugs. This type of “penny pincher” mentality is not what I’m advocating for. Rather the intention is to realize that being cost-conscious isn’t intrinsically aligned with our daily tasks as cloud and software engineers. But if we pick up some infrastructure-centric cost savings habits and voluntarily align ourselves with dollar savings KPIs, it can be of great benefit to us at any stage of our career.

Since career stages and titles are many times company-specific and ever-changing, for sake of simplicity, I’ll break down how we can benefit from a cost-saving mindset at these three rough levels:

• Junior
• Senior
• Manager

The Junior

Keyword: proactivity

As a junior developer or engineer, you are probably just trying your best to not drown and keep up with what you have coming at you already. When you start feeling more comfortable in your role and the imposter syndrome starts to lose its grip on you, start getting proactive.

I remember researching what it took to transition from a Junior to a Senior position in my first year as a DevOps engineer. I stumbled upon a Pelado Nerd video (Spanish content) in which he said something that stuck with me. He said, “do the job of a senior before you have the official title”. He goes on to state that from a junior role you should do more than what you are asked. Not necessarily grab more tickets and do more work, but rather, go the extra mile.

Going the extra mile

There are many ways to do this. Depending on the tasks, it might involve updating documentation or creating bug tickets when you see them or being one of the first to help when others need assistance. But, without a doubt, one of the best ways to stop being seen as a junior is to adopt a cost-optimization mindset.

Here are a few things you can do as a junior:

• Start by monitoring critical metrics (resources usage, incoming requests, errors rate, etc) of the services/apps you’re building or depends on (databases, APIs, etc)
• Check the utilization of the VMs your workloads run on, check if they are clearly underutilized. If they are, bring it up and try to find out why.
• Check if there is any legacy or unused objects running in testing or pre-prod Kubernetes clusters that can be removed.
• Remove completed Kubernetes jobs.
• Ask your manager which are the recurring 5 most expensive items in the organization’s cloud bill and focus on those.
• Go online and find a cloud optimization checklist article and start applying it to your cloud environment.

The upsides for junior engineers are considerable. Internally, your teammates and boss will see your proactively bringing undeniable value to the team. But it’s also externally positive since any skills you pick up in the process can be easily applied to any future position.

The Senior

Keyword: leadership

Senior engineers can undoubtedly benefit on a personal level the same as juniors. Additionally, engineers at this level of seniority can set the bar for other less experienced teammates and greatly influence the habits newer generations pick up.

I, myself am a professional who has been enormously impacted by a handful of legendary senior engineers in my career. I can say for a fact that when we are young and hungry, we are sponges and will soak up the positive traits of those around us. It reminds me of why they say that it’s important to learn to drive from a professional instructor and not your dad since the instructor probably won’t pass on bad habits.

I get it if you have no interest in being a mentor, that’s fine. But it’s undeniable that at this stage in your career you can choose to have a huge impact by being the instructor, you can become a legend.

Another way to bring value is by working to improve cost visualization. By bringing granular and custom cost dashboards to the team, you will be creating a solid base for less experienced team members to find a place to start, and your own effort will be much more quantifiable. Through clear dashboards, savings KPIs and objectives can be set and measured. You can accomplish this by using a tool like Tailwarden which has a comprehensive cost breakdown dashboard alongside many helpful features like the Inventory page that can enable you to search, filter, and tag your resources in one place as well as track price fluctuations. Enabling you to make informed optimization decisions from one place.

It also goes without saying that if you are ramping up for your next performance review and you are planning on asking for a raise, it won’t hurt to have hard data to point to that speaks to how much money you saved the company.

As engineers we are usually thought of as value adders, let’s expand the thought to value-savers too.

The Manager

Keyword: transparency

As a manager, you want to be able to transfer the incentive of cost reduction down to the individual member level so that your team can feel incentivized to make a difference. It seems logical to build rituals and openly celebrate individuals who take the time to not only do their job but also to do it in a way that saves money.

If saving tech resource dollars is important to you it’s crucial to be transparent about where the money is going in the first place.

I remember working at the same company I mentioned above, I was tasked with bringing down the cost of a proprietary APM monitoring tool that was sending an increasingly expensive bill month by month. It was only when my manager handed me a detailed breakdown of the bill that I was able to bring it down from 20.000$/m to just over 8.000$.

Regarding the rituals that would be beneficial towards incentivizing a cost-conscious culture. Consider giving the team a taste of the action if significant savings are achieved. This doesn’t necessarily have to be a financial incentive but perhaps make an event out of it, name team cost heroes, and get some pizzas. It’s up to you really, but make it known that your team is valued for their work.

Enable your team members to be creative when it comes to finding the best way to visualize the data you made available to them and when they deliver, recognize them for it.

Conclusion

Cultural phenomena aren’t limited to certain experiences or status levels. People from every social class and background enjoy football and paella in Spain and most Americans will touch some grass and have a cold beverage on the 4th of July. Culture is something that takes a while to take root but once it does, it’s something that every person on the team can participate in and can bring people together. Regardless of where you land on your company's organizational map. If you focus at least 2%-3% more of your time on being more cost-conscious, I’m sure you will reap the fruits.

Regardless if you are a Developer, DevOps, or Cloud engineer, dealing with the cloud can be tough, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden Discord server. Where you will find tips, community calls, and much more.