Lessons

Setting up a S3 Bucket with Cross Region Replication

In this tutorial, I'm going to show you how to setup an S3 bucket. There are lots of aspects to an S3 bucket and this tutorial only scrapes the surface. We will put a file in the bucket, retrieve the file, and delete the file. We will also briefly go over IAM Roles, IAM Users and bucket policies.

Setup

Start by heading over to Amazon Web Services (AWS), and signing in. Under service in the top left corner, you should see S3. From the S3 homepage, hit Create Bucket.

You will need to provide a unique bucket name. Bucket names are global so you cannot have the same bucket name on both east and west. I recommend adding the region as a suffix. In step 2, I enable server side encryption by default. This keeps the files at rest encrypted unless they are retrieved.

Your bucket is all setup and you should be able to see it in the list of buckets.

If you are interested in cross region replication (files are copied to another bucket in a different region), you will need to setup a secondary bucket. Repeat the steps above but switch regions on step one. If this doesn't interest you, move on to the next section.

IAM Roles & Users

I recommend reading through AWS documentation on IAM Roles and Users.

At a high level, IAM Users are typically tide to a user (Ex. Me). An IAM role is tie to an application or a role of a user. You own a company with two developers: John and Sam. John and Sam both have IAM users associated with them. You can even allow them to login using this IAM user.

Your first option: We can setup their IAM users to have access to our bucket. This would be through the bucket policy and their IAM users having S3 permissions.

The second options: There can be an IAM role called Developer for example. This role is the set of permissions each developer at the company gets and it's attached to their IAM User.

There are many cases for this. The last common use case, you spin up an EC2 with an instance profile (IAM Role). This role can be added to the bucket policy to give the EC2 permission on the bucket.

IAM User Creation

We are going to create an IAM User only to demo getting access to the bucket. Under the services tab in the top left corner, open IAM. Select Users on the left and click Add User. You'll have to name the role. I am only going to give programming access but you can set it up so your developer can log in using this IAM User.

We will attach permissions to the IAM role. For this demo, I'm just going to attach the S3 Full Access that is managed by AWS. I would recommend reviewing the permissions that are need by the user.

Hit next, next, and create user. You will have the credentials on screen, save them. Save the access key id, and the access key secret.

We should be all setup for the IAM User.

Bucket Policy

We need to create a way to manage who has access to our bucket. For this, we will use the S3 bucket policy. It can be found under Permissions then Bucket Policy. It starts empty.

We can use the policy generator tool to create a new bucket policy. We can add both Allow and Deny. We can specific what users have access on which content, and more.

The principal is what you want to have access or not have access to a specific resource. If you put * with an allow, any IAM role could access this resource. If you put * with a deny, then everything will be denied.

The principal will be the IAM User ARN. This can be found on the IAM Users page, under the details of the specific role.

The other piece of information required is the resource. I am going to create an allow policy on my bucket. I will need the bucket ARN. This can be found above the bucket policy textarea.

I put both in the generator, I will select the following actions:

1
2
3
PutObject
DeleteObject
GetObject

Hit add statement then generate policy. My policy looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
  "Id": "Policy1568839482235",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1568839480768",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-east-1",
      "Principal": {
        "AWS": [
          "arn:aws:iam::my_aws_account_number>:user/DemoS3BucketUser"
        ]
      }
    }
  ]
}

I have an actual value for my_aws_account_number>, it was removed after.

I'm going to modify this slightly. This:

1
"Resource": "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-east-1"

Gives you access on a bucket level, compared to:

1
"Resource": "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-east-1/*"

This above gives access on an object level. This will stop bucket actions like ListBucket, but since I only have object based actions this is okay. You can also do this:

1
2
3
4
"Resource": [
  "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-east-1",
  "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-east-1/*"
]

Paste your policy in the bucket policy textarea and hit save.

We can test this out. There are a few options, you can setup the AWS CLI. You can manually add your credentials in ~/.aws/credentials and run commands on the bucket. I will use Python 3 for this demo.

Another option is to create a Python3 script, with touch test.py. You will need the boto3 library. It can be installed with:

1
pip3 install boto3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import boto3

def main():
    session = boto3.Session(
        aws_access_key_id="your_access_key>",
        aws_secret_access_key="your_access_secret_key>",
    )
    client = session.client('s3')

    response = client.put_object(
        Body='test',
        Bucket='demo-sample-bucket-name-from-keithweaver-us-east-1',
        Key='folder1/myfile.txt'
    )
    print ('Done uploading')


main()

This is a very simple application. You can run it with:

1
python3 test.py

It will upload a file containing the words test.

As you can see it was successful. You can try working and playing around with the S3 bucket.

Cross Region

Next, we are going to setup cross region S3 bucket replication. We will setup a new role manage this replication. We should not use a user for this. Head back to the IAM page. Select Roles on the left side. Select Create Role. Select S3.

Again, I'm going to select FullAccess but you should limit it to the access the IAM role needs.

Select next to tags, then review. Give the IAM role a name.

Your role is all setup. Now, it needs to be added to both bucket policies. Your replication role needs the following actions:

1
2
3
ReplicateObject
ReplicateDelete
ReplicateTags

My us-east-1 bucket policy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
    "Id": "Policy1568839482235",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1568839480768",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-east-1/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::your_account_number>:user/DemoS3BucketUser"
                ]
            }
        },
        {
            "Sid": "Stmt15688394801231",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-east-1/*"
            ],
            "Principal": {
                "AWS": [
                    "arn:aws:iam::your_account_number>:role/DemoSampleBucket_Replication"
                ]
            }
        }
    ]
}

My us-west-1 bucket policy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
    "Id": "Policy1568839482235",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1568839480768",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-west-1/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::your_account_number>:user/DemoS3BucketUser"
                ]
            }
        },
        {
            "Sid": "Stmt15688394801231",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::demo-sample-bucket-name-from-keithweaver-us-west-1/*"
            ],
            "Principal": {
                "AWS": [
                    "arn:aws:iam::your_account_number>:role/DemoSampleBucket_Replication"
                ]
            }
        }
    ]
}

Next step is to setup the replication. We can do this under the Management tab. Select Replication. Hit Add Rule.

You may receive this screen. Enable versioning.

Select how much you want to replication. I will replicate the full bucket.

Select the destination bucket. The destination bucket should have versioning also enabled.

Select the IAM role and give the rule a name.

Save your new rule.

Repeat the steps for the other bucket as well.

You can test this using a similar Python script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import boto3

def main():
    session = boto3.Session(
        aws_access_key_id="your_access_key>",
        aws_secret_access_key="your_access_secret_key>",
    )
    client = session.client('s3')

    response = client.put_object(
        Body='test',
        Bucket='demo-sample-bucket-name-from-keithweaver-us-east-1',
        Key='east_to_west.txt'
    )
    response = client.put_object(
        Body='test',
        Bucket='demo-sample-bucket-name-from-keithweaver-us-west-1',
        Key='west_to_east.txt'
    )
    print ('Done uploading')


main()

Run the script:

1
python3 test2.py

As you can see, one file has the file replication status of REPLICA, and the other COMPLETED. Next, check the west bucket. It should look the same.

Conclusion

S3 buckets are powerful tools whenever you are building applications.

There are a number of security aspects I skipped like deny policies, settings access to follow least privilege model, harding coding credentials and more. I would recommend reading the AWS documentation on both IAM and S3. This was simply to get you up and running.

Your buckets are all set up! Thanks for reading.