We think black boxes should be left in planes and out of tech.

The original black box

The first thing that comes to mind when the term black-box pops up in conversation is usually the well-protected recording device that every modern plane is equipped with to record all cabin communications and the onboard system metrics. If there is a plane crash, apart from locating survivors, retrieving the black box is always the top priority. It holds crucial information and tells the story of the sequence of events that leads up to a crash. By understanding what went wrong in every case, lessons can be learned and future catastrophes avoided.

It was through the book Black box thinking by Matthew Syed that I fully understood what the black box represents in aviation on a larger scale. It represents a system that is deeply committed to learning from failures and open to feedback (for the most part). To learn from mistakes and to fix known issues so they don’t happen again. Aviation is one of those industries that improves over time due to its embrace of an open feedback loop. Whereas others, such as the medical industry are unfortunately still very reluctant to accept and learn from mistakes and embody the opposite approach, the closed feedback loop.

An easy way to think of these two types of feedback loops is by trying to get better at golf. By practicing during the day, with each hit you can adjust your shot to get the ball closer and closer to your objective. Since you can see the result of your action you can adjust and improve. But imagine if you were practicing in the dark. You don’t see the result of your action and can’t see where the ball landed, leading you to not have the information you need to make shot-improving adjustments. That is a closed feedback loop.

The inverted black box

In the tech industry, on the other hand, the term black box when used to describe a product has a different connotation altogether. It usually makes reference to a tool that promises to automate away all the work you don’t want to do. Magically doing things that would usually take a team of people to perform. With just a push of a button, you are up and running and you can sit back and relax.

The selling point is usually based on the premise that what you are trying to do is pretty complex, so just let the black box tool take care of your problem and abstract your headache away in exchange for less freedom and some defined guardrails you will have to adapt to. The main problem with this approach that might not be apparent at first glance is that by depending on such a tool you are walking straight into a closed feedback loop.

As the saying goes:

Give a man a fish and he will eat for a day, teach a man to fish and he will eat for a lifetime.

Life as cloud engineers, developers, project managers, and CTOs is difficult as it is. We already have to put so much time into learning so many styles of “fishing”. We have to adapt to new web development frameworks, learn new coding languages, and stay on top of the latest best practices. So I understand the temptation of wanting to give in and hand over responsibility to a third party to avoid over complications. But there is no denying that creating a dependency on tools that simply “deploys your app in an instant” or “provisions and manages your cluster” without your help is just putting a band-aid on without understanding the root cause of the ailment.

If your child was afraid of sitting an exam, would you pay for somebody to take the exam for them? Or would you bring in a qualified tutor to help them in the areas they need and prepare them not only for the exam right in front of them but for the many other exams they’ll face down the road?

Try to avoid tools that do great things “under the hood” and focus more on tools that break down and simplify decisions and build up your knowledge to make you the ultimate decision maker and not a passive spectator.

Avoid the red flags

The cloud space is full of miracle tools and all-in-one sensations. It is not surprising because cloud providers offer such a plethora of managed services and possibilities that if you are not experienced or know what you are doing, it’s easy to get lost and make mistakes that impact your productivity and hurt you financially.

Well funded teams with bountiful resources can hire cloud experts and specialists who are extremely experienced and well equipped at getting the most out of the cloud. They make sure the organization stays nimble, using the most appropriate and well suited tools for the job in each case. They can get creative and tether cloud providers together, adapt architectures, and flexibly adapt as business needs change. This is the dream.

But of course, this is where the problem is. Not every organization has limitless resources to spend on dedicated cloud specialist teams, that is if you even find them in the first place. This is where the appeal of tools that claim to be able to automate that work comes in. There is no shame in accepting your situation and trying to do the best with what you have. But no matter which fully automated cloud tool you choose, it will come with a series of trade-offs that might come back and haunt you.

Firstly, you are locked into the cloud provider the tool has chosen. You are limited to the services the tool has made available. How frustrating it must be if AWS launches a new EC2 instance version that is half the price, but you can’t use because the tool didn’t add it to it’s service catalogue. Cloud providers are constantly adding and launching new services, it’s very unlikely that the tool will be able to keep up.

You are as flexible as the tool allows you to be. If your business grows in a way you weren’t expecting, how do you know you will be able to scale in the most appropriate way? There are certain decisions that are now not only up to you and your abilities. For example, if you want to fully change your tech stack or codebase, you will have to first check if the solutions allows for it. What will you do if the tool stops supporting a key service in your infrastructure or unexpectedly goes out of business? If you stop using the tool in a year but you have another app you want to deploy, will you have learned how to do it?

Another concern is around data protection, as a company in the EU for example you need to comply with GDPR meaning you need to ensure that personal data is stored in the EU and also any data processing is done on servers in the EU. By fully outsourcing these tasks, you cannot fully control where the data is processed or stored. And if the third party tool breaches any GDPR regulations the responsability is still yours.

The list of cons can go on and on, whereas the list of pros seems pretty thin.

Bet on tools that help you grow

Tools that are more open, expansive, and flexible are not diamonds in the rough that are impossible to find. You just need to know what you are looking for. At Tailwarden, we have a pretty ambitious mission which is to help the 99% of developers to take control of their cloud infrastructure. We don’t want to take care of it for you which is the key distinction.

We aim to help you take control of the different aspects of the cloud such as cost, security, and performance. For instance, we offer a consolidated cloud agnostic inventory page that pools all of your resources across accounts and providers. Allowing you to tag and create in-depth cross-sectioned cost breakdowns. We help you stay under budget by setting cloud account budget alerts that trigger notifications when surpassed. Through the soon-to-launch IAM analyzer feature, we will allow users to edit IAM roles that might constitute security vulnerabilities and help follow best practices by editing problematic policies right from the Tailwarden dashboard.

We have a long list of features coming down the line as part of the roadmap such as custom dashboards, data slicing with custom filters, and expanded team management functionalities. We are constantly expanding the roadmap to grow the platform into a tool that can really make an impact on the life of every developer. Even though each feature will be different, one thing they will all have in common is that they will always strive to increase open feedback loops, helping the user stay informed and aware of the best way to manage their infrastructure.

Lastly another key aspect of these types of tools is that they usually have a vibrant community surrounding them. In the case of Tailwarden we are lucky to have a growing group of users scattered across the Discord server (Join us!) and GitHub that participate and offer ideas on how to use, expand and debug issues when they arise.

A final thought

Hopefully, with open feedback loop promoting tools like Tailwarden we can do our part in forwarding the message that developers don’t need things done for them, they just need tools that make it easy to learn new and difficult things. One step and one feature at a time, we aspire to not only help the developers of today but also shape the developers of tomorrow.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden Discord server. Where you will find tips, community calls, and much more.

‍