Are you sure you own your data?

Whoever owns the keys to your data, owns your data. In many cases that is not you.

When you took up AWS for the first time, it felt confounding for all of its options. You picked that single feature you needed and realized how powerful it was. Soon, you built your whole development environment on it, and once your managers and/or founders noticed how far you had gotten by yourself, you ended up having a production system running on it as well.

Consulting our customers, we see more often than not that AWS environment is set up organically. It has been used to solve simple issues, and over time more and more features have been included from the AWS offering. While this is exactly what AWS is good for, and its cost model supports incremental steps perfectly, it is often dangerous from a security point of view.

The most common access-related threats we see are:

  • Too high privileges to begin with
  • No time to review and restrict access policies
  • No separation of duties
  • Convenience over security for developers and admins.
  • The root account owned by an external company.

Security gone wrong

A cautionary tale of AWS security gone wrong is Codespaces.com, a SaaS cloud storage provider. I should say was, because after their AWS account was breached, they went out of business shortly afterwards, unable to recover their customer data.

What happened to Codespaces was that an attacker gained access to their AWS root account, set up a denial of service and demanded payment to return the assets to Codespaces. Codespaces staff tried taking control of their account, but failed. Seeing this, the attacker proceeded to wipe everything. All the data, all the servers.

Codespaces posted a letter of apology shortly afterwards, informing their customers they could not recover the data, and not being able to pay their customers reparations, the company shut down.

Privilege levels you should identify:

First step in securing your system is to reduce the user authorizations bare minimum that they need to perform their tasks. A good starting point for this is classifying your AWS privileges into different severity levels by purpose and storage method.

 AWS account root: manages everything, owns everything, is the nuclear threat to your organization. Set up complex password, set up hardware MFA, remove access keys, set up CloudTrail at root access only bucket, set up administrative accounts and lock away the key to this account.
 Administrative console account: Full rights to operational aspects, can not terminate account. Complex password, at least virtual MFA, no access keys. The main role of the admin users is to create limited IAM users with access rights for specific application requirements. Once you have created the limited user for your current need, switch to using that account.
 Secure host access keys – including localhost. Only to be used on machines that you are certain that will not be exposed. Should have all the IAM privileges needed to run their role, including potential deletion. Consider requiring MFA for deletions. For purely machine used access management you should use IAM roles to control access. Not only are you relieved from key management chores, but the recently released managed policies make Role access control even easier than before.
 Public access keys. Access keys can be handled in plain sight if necessary – in fact for javascript usage it is the only way. In this case, you should ensure that your user has the minimal privileges to specifically named resources, for which you understand the possible effects of malicious actions. For certain services (S3, for example), you should also use request signing to further restrict requests. If you do have a backend, client side access to non-S3 resources should be handle via temporary credentials from STS. Key takeaway for public access credentials is that they should be very limited, thoroughly considered but that they can enable very powerful features.

Convenience over security

Recently I have come across several instances of root keys being managed by external company, most usually your cloud consulting partner or development partner. There is absolutely no reason to do this. Every operation that you can do with root account, you can authorized IAM user account to do as well, including viewing billing information.

The main reason you need to hold on to the root account keys is that only the root user can be absolutely certain that your CloudTrail logs have not been tampered with. It is also the only user that can not be deleted or changed at whim of any Administrator-privileged user.

Auditing

For many companies the effort to educate their admins on the finer points of AWS security may be too much, since everyone is overloaded with work. In this case I recommend buying the expertise from outside – find your local AWS consulting partner and ask for a quote on security audit of your environments. At the very least, require that they check for the issues detailed above.

Conclusions

More widespread the AWS usage is in your organization, more important it is that your operational security is up to task. Strict privilege management prevents unintended side effects and mitigates damage potential in the case of your keys being compromised.

Key management should always be ultimately in your own control. Regardless of who you have running your AWS operations, the account root keys should be locked away behind the top responsible tech officer, with CloudTrail set to follow every action.

With the root access in your own sole control, you can still give your contractors and new employees admin level access without compromising your audit trail and ultimate control over your own operations.

Security requires continuous improvement, tuning and reviewing to stay up to speed in an environment where setting up a massive customer facing service could take less than 15 minutes to create. Well-thought privilege restriction and periodic auditing of your whole system are the practices every production system owner should have in place.

Jukka Dahlbom
Jukka Dahlbom
Head of Data Engineering, Co-founder Tietotyön vastapainoksi vaellan, musisoin, tanssin ja pelaan.