Some define architecture as the decisions that are expensive to undo. Well, here's an excellent candidate: your cloud account, security and networking strategy.

I recently led the WebomateS team through a massive refactoring, blowing up one AWS account and a flat network, and starting over with three accounts, segmented networks and centralized sharing of IAM users, AMIs and ECR repos. Maybe you'll benefit from my retelling.

Motivation

We had three strategic goals in mind:

1. Segment (isolate) lower and upper environments: dev, qa and prod
2. Segment cost activities in each environment, but also enable consolidated views
3. Segment network into public and private subnets

And our constraints:

1. Minimize AWS costs
2. Minimize operational overhead

In Part 1, I discuss the issues in account linking and sharing. Part 2 goes details the networking design.

Possible Solutions

We considered two strategies to isolate activity, but also enable simple promotion of release candidates. Note: At this stage, we're single region (hopefully not for long!), so we only need one VPC.

First up, a single account, with one VPC per environment:

Second, multiple accounts, one for each environment:

Multiple accounts wins as it provides better isolation and cost segmentation. Fortunately, AWS has engineered all the mechanisms we need to centralize stuff in one account and share it to the others. This includes:

• Identity & Access Management (IAM) users
• Docker Images in Elastic Container Registry (ECR)
• Amazon Machine Images (AMIs)

In our case, we chose the dev account, since the resources get built and tested there first. You can put users wherever you like, and even create an dedicated Identity account, as this AWS Multi-Account Security Strategy article describes. I thought it simpler to keep them in dev. Also we have many more users that may access dev than in qa and even less in prod.

Each account has its own dedicated compute, storage and services to support activity (hopefully duplicates, at least functionally), and roles and policies to control access.

By the way, none of this sharing involves any extra costs, not even data transfer charges (assuming all accounts are in the same region).

IAM

Let's look first at IAM user sharing for both Single-Sign-On (SSO) AWS console access and for AWS Command Line Interface (CLI) access.

Users are put into one or more groups along job function, for example: dev, dev-leads, qa, ops and billing. We also create corresponding 'cross-account' roles in qa: cross-dev, cross-dev-leads, cross-qa, etc. Likewise in prod with the same names.

The groups are granted:

• Dev account permissions via AWS or Customer Managed IAM policies
• QA and Prod account permissions to switch/assume an appropriate role

The cross-account roles are granted:

• Account local permissions via AWS or Customer Managed IAM policies

Ops permissions are rather similar across accounts, but developer permissions go in a pyramid shape. QA and Prod access is quite limited and only there for triage. The QA shape depends on your organization's QA process.

Switching between accounts is super easy. Once it's set up, it's a single menu selection (two clicks) to switch.

Note that above I show App or Software users. I recommend avoiding those if you can. We are migrating our apps away from access/secret keys for AWS service access to EC2 Instance or ECS Service roles, which are credential-less.

AMIs

AMIs may be shared to other accounts on a per AMI basis. These are made available to all account IAM users, EC2 Instance and ECS Service roles once granted. You can simply reference them in Individual EC2 Instance Launch, [Scaling Group] Launch Configurations and CLI access. They appear as 'Shared With Me' in the wizards.

ECR

Like AMIs, ECR repos may be shared to other accounts on per repo basis, and again to all users and roles in that account.

VPC

None of the IAM, AMI or ECR sharing requires any cross-network access, however you will need VPC Peering Connection if you want to enable it. (In Part 2 I'll discuss why we peer to save costs on infrastructure).

Note that AWS Peering Connections are direct, one-to-one connections, so three are necessary for three VPCs, six for four VPCs, and so on...

Part 2 will discuss firewall pinholing, Network ACLs and such.

Billing

We can view costs per account or consolidate to one view via the Master account, in our case the Master, since it was first, with the AWS Orangization construct. Again no extra costs to join accounts.

Your CC is charged from the Master account.

Summary

Splitting into three AWS accounts achieved our goals of segmentation of activity while minimizing both cloud provider and internal operational costs.

And, although it was a lot of effort, the reboot enabled us to normalize access control mechanisms, both in IAM and Networking (Security Groups), leading to improved security and operations.

Finally, getting your environment strategy set before you start building (especially networking) is ideal. But if not, no worries - you can always phase in the migration.

---

Published Jan 8, 2018
Cloud icon made by Yannick from www.flaticon.com, CC 3.0 BY license
Comments? email me - mark at sawers dot com