Uncomplicating cloud Security - Detection (Part 3)

One of the earliest memories I have as a child is going for a drive with my dad in our old grey Opel Corsa. When it was time to go home, we would pull into our street and try to find a parking spot. I didn’t live in a bad neighbourhood per se, at least not to my knowledge. But even so, before locking the car my dad would always walk around to the boot and grab a bulky steering wheel clamp and click it into place immobilizing the steering wheel. I remember the object vividly, it was bright yellow, probably to let potential thieves know exactly what they would be up against if they tried to rob the car. I remember how heavy it was too, the struggle to lift the clamp up a time my dad let me try to install it was real, granted I was a child but still. Only when the lump of fluorescent yellow metal was in place could we rest assured that the car would most likely not be stolen in the night.

I had completely forgotten about this relic of my past until recently when I started thinking about what instruments and mechanisms we had when systems of detection weren’t that sophisticated. Nowadays, cars have anti-robbery alarms that can notify an app to let you know about any funny business going on, home alarms have integrated systems to notify the police station if an entry is being forced into your home. Your smart home smoke alarm can notify the fire brigade on your behalf, so you can just focus on getting to safety in case of a fire.

Undoubtedly with more sophisticated systems of detection, we can change and evolve how we protect our assets in the first place. Thankfully innovation and the development of better alarms have suppressed to need to carry these 5-kilo single-function contraptions.

The same goes for the world of the cloud, mechanisms of protection that worked in a datacenter have to be expanded to understand the context of the cloud. We don’t have access to our physical servers anymore, so we can’t just slap a figurative clamp on them and think we are protected. It’s important to understand the many ways we can encounter malicious behaviour and know what to do once we catch wind of it.

To know how to detect a malicious actor is to know which clues give us the most damning information. If there was an attempted robbery on your car, one of the first things the police officers on a scene will look for are fingerprints or DNA. In the cloud world, we look at the logs.

Where, How, and Why Enable logs

Security breaches can happen, if anything, we should face the possibility as more of a question of when than a question of “if” our cloud account will be attacked. If it’s the case that we have been breached and there is some investigating to do we need to have a clear idea of what resources we have at our disposal. As security-minded professionals, your most trusted allies are your different sources of logs. Each service can be enabled to store logs and with a thorough inspection of them, we can piece  together the series of events and find the source of any incident. Let’s look at the logging services or log adjacent services you should care about, if they are enabled by default and their possible use cases.

Logging services

I have the logs, now what?

Logs aren’t that valuable unless you inspected and derive insight from them. We have a multitude of options at our disposal to try and get the most of our logs when we need to the most, we can use services like AWS Security hub, AWS Athena to query CloudTrail trails stored in S3 buckets or even better, CloudTrail Lake.

AWS Security Hub

Provides a single place that aggregates, organizes, and prioritizes your security alerts, or findings, from multiple AWS services and optional third-party products to give you a comprehensive view of security alerts and compliance status.

• Enabled by default: Yes
• Pricing: Free trial, then charged by security check done
• Use case: Security hub can be considered a multitude of tools in one. You can conduct Security Posture Management (SPM), and leverage EventBridge to initiate security orchestration and automation workflows. Easily integrate streamlined data ingestion with downstream tooling. Above all, have a centralized place to correlate security findings to discover new insights.

CloudTrail Lake

Traditionally the recommended way of querying CloudTrail events would be to create trails and store them in S3 buckets, querying the logs using AWS Athena. We can now use CloudTrail lake instead which allows for richer and faster SQL-based queries directly in the CloudTrail console, no need to use Athena for the job.

Additional benefits of using CloudTrail lake is the fact that you can save previously run queries for future use and also save the query result which spans cloud accounts and regions in an S3 bucket. Once the data store is set up, it’s time to run some queries, conveniently we can do so right in the console using the query editor.

AWS gives us access to a bunch of sample queries to run, such as: a query to find the users who signed in to the console the most within the past week.

If you would like to go a little bit deeper and explore CloudTrail logs more in depth setup an ELK stack in your AWS account and integrate it with an S3 bucket that you are sending a CloudTrail trail to.

Visualize and inspect CloudTrail with ELK stack:

If you want to use a battle-tested fully customizable log management implementation, the self-managed logs storage and visualization ELK stack (ElasticSearch, Logstash, and Kibana) is a great way to save the cost that you would incur by using one of the centralized logs management tools mentioned above. You can find a huge amount of documentation and tutorials on how to use these tools in combination to make some highly useful and insightful dashboards.

The implementation consists of the creation of a CloudTrail trail which is stored in an S3 bucket which is then integrated with logstach which reads and parses the logs which are then sent to be indexed in Elasticsearch and are then made available to be visualized through Kibana dashboard

Here we can see an example of a Kibana dashboard which shows a map of the regions that our AWS account is being accessed from:

Deploy it yourself to your own environment following this straightforward tutorial written by Tailwarden CTO, Mohamed Labouardy.

Conclusion

It’s never a good feeling to be hacked, it’s even worse when you look around and inspect the damage and you realize that there were multiple things you could have done to stop it and there were tools available to help you detect malicious or negligent practices but for some reason you never got around to using them. Don't let this be your case.

A good way of thinking of detection is by understanding the two layers we have to work with. We have the logs gathering layer and we also have the logs extraction layer (making sense of what the logs show)

When it comes to collecting logs, we always want as much as possible so be sure to enable some of the non-default logs such as ELB, VPC flow logs, RDS, and S3 access logs if you are using them in your accounts. Having these logs will come in handy when you need to trace and stitch together actions to build a full picture of what is going on in your environment. You could take it upon yourself and try your best to comb over and make sense of the logs on your own, depending on the number of logs this could be a very time-consuming and cumbersome task. Move into the second layer of action and utilize the tools mentioned above such as AWS GuardDuty and Amazon Security Hub or even a self-managed ELK stack and make sure that no rock is left unturned and that you are aware of what is going on in your AWS Organization at all times.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden discord server. Where you will find tips, community calls, and much more.