Amazon web services with boto3 library

introduction

Amazon web services (AWS) is a useful tool to alleviates the pain of maintaining infrastructure. It makes requesting cloud computing resources as easy as either clicking a few buttons or making an API call. Also the price is quite affordable even for individuals. You can estimate the monthly cost based on approximate usage with this page.

The main components of AWS include

• computing
• storage
• networking
• database

Take a web application for example, the most relevant resources would be Elastic Compute Cloud (EC2), Simple Storage Service (S3), and Relational Database Service (RDS).

Currently AWS provides 1 year free trial to new users. See the AWS free tier page for details.

So far I find the official AWS documentations not so user friendly: it is like an encyclopedia that includes all kinds of topics but gets sketchy on details. In this post I will share my notes for the following tasks:

• basic account configurations
• spin up and down on-demand or spot EC2 instances
• ssh to EC2 instance
• access S3 from EC2 instance without credentials

Both AWS web console and python boto3 library are used for these tasks. The web console is convenient for configurations and the boto3 API is good for automation. For the web console part, I will only jot down what needs to be done and possibly include a link to the relevant documentations.

basic account configurations

After signing up for AWS (free tier account for example), the first thing to do is to set up a user account. This enables you to access the AWS resources from your own computer with either the command line interface awscli or API calls. Without it, the only way to access AWS is to log into the AWS web console.

There are three steps to set up the user account:

1. create individual IAM user, e.g., dev
2. create user group to manage permissions, e.g., dev-group
3. create local credential files

The first two steps can be done via the aws web console. To start with, we can attach the following permissions to the user group

• AmazonS3FullAccess
• AmazonEC2FullAccess
• AmazonRDSFullAccess

To set up the local credential file, you can use the awscli command line tool. To install it and the boto3 library, run

pip install boto3, awscli

After installation, run from your terminal

aws configure

It will ask user input for AWS Access Key ID, AWS Secret Access Key, and other preferences. They can be found on the aws web console as you create the account. After execution, two files will be created locally. The AWS Access Key ID and AWS Secret Access Key are stored in ~/.aws/credentials, which looks like

[default]
aws_access_key_id = your_id
aws_secret_access_key = your_key

Other preferences are stored in ~/.aws/config, which looks like

[default]
output = json
region = us-east-1

To use multiple user profiles, you can edit the credential and config files directly. For example, the credential file could look like

[default]
aws_access_key_id = your_id
aws_secret_access_key = your_key

[admin]
aws_access_key_id = admin_id
aws_secret_access_key = admin_key

spin up and down EC2 instances

There are two types of EC2 instances one can request

• on-demand instances
• spot instances

The main difference is in their pricing models. For on-demand instances, you pay a fixed price up front with fixed rate. On the other hand, you provide a bid price for spot instances, and the usage is charged at the market price as long as it is lower than your bid price. When the market price exceeds the bid price, the running instance will be killed with a two-minute courtesy window.

The spot instance market price is usually much cheaper than the on-demand price. For example, I just took a peak at the US-East (N. Viginia) m4.large price. The on-demand price is $0.1 per hour and the spot market price is $0.0274 per hour.

The easiest way to spin up or down an EC2 instance is via the web console. If automation is needed, boto3 API can be used.

• boto3
  • session
  • config
  • resource: high-level object-oriented API
    • identifiers and attributes
    • actions
    • references
    • subresources
    • collections
  • client: low-level access to Botocore

There are two sets of APIs in boto3, the so-called resource and client. The client API provides many more functionalities than the slightly more user-friendly resource API.

To spin up an on-demand instance, run

import boto3

s = boto3.Session(profile_name='dev', region_name='us-east-1')
ec2 = s.resource('ec2')

rc = ec2.create_instances(ImageId='ami-4fffc834',
                          InstanceType='t2.nano',
                          MinCount=1,
                          MaxCount=1,
                          )

Here the ImageId is the disk image to load for the instance, i.e., the so-called Amazon Machine Images (AMI). It contains the operating system and pre-installed libraries. The one used here is provided by amazon and you can make your own to suit any special need.

To spin up a spot instance, run

s = boto3.Session(profile_name='dev', region_name='us-east-1')
client = s.client('ec2')

rc = client.request_spot_instances(
                DryRun=False,
                SpotPrice=str(price),
                Type='one-time',
                LaunchSpecification={'ImageId': 'ami-4fffc834',
                                     'InstanceType': 'm4.large',
                                     },
                InstanceCount=1,
               )

where price is the bid price.

To kill the instance, run

s = boto3.Session(profile_name='dev')
ec2 = s.resource('ec2', region_name='us-east-1')
ec2.Instance(instance_id).terminate()

where instance_id can be looked up either from the aws web console or the awscli.

ssh to the instance

In order to ssh into EC2 instance, you need to assign to the instance

• an aws key-pair when launching the instance
• a security group with ssh permission

To check whether these two conditions are met for your instance, go to the EC2 Management page of the aws web console, click on your instance at the instance tab, and check if there is a Key pair name associated with it, and whether the Security groups inbound rule contains port 22 tcp protocol.

When both conditions are met, simply run

rc = ec2.create_instances(ImageId='ami-4fffc834',
			  InstanceType='t2.nano',
			  MinCount=1,
			  MaxCount=1,
			  KeyName='my-key',
			  )

to create the instance. Here I omit the SecurityGroupIds argument since it defaults to the one I set up.

To ssh to the instance, first you should download the credential file from the aws web console. Then modify its access permission by

chmod 400 /path/my-key-pair.pem

Then run

ssh -i /path/my-key-pair.pem ec2-user@1.2.3.4

where the ip address can be looked up from aws web console. Depending on the image you load, the user name could vary. Possible ones include ec2-user, centos, ubuntu, root, etc.

access S3 from EC2

Accessing AWS S3 is somewhat similar to accessing a FTP server.

• S3
  • buckets
  • objects: 5TB data limit
    • object data
    • metadata: name/value pairs
  • permission
  • keys
  • regions
    • US East (N. Virginia)
    • US East (Ohio)
    • US West (N. California)
    • US West (Oregon)

Data in S3 are organized by the so-called buckets. From the command line, you can list the content of a bucket by running

aws --profile=dev s3 ls s3://my-awesome-bucket

Here my-awesome-bucket is a made-up name.

To grant S3 access for the EC2 instance, there are two things to do

• for the instance, create IAM role with the IAM policy for S3 bucket access
• for the user profile (dev in my case), grant permission to pass IAM role to instances

For example, a simple policy to grant read/write/list permission to a bucket called my-awesome-bucket for the EC2 instance looks like this

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:ListBucket"
 ],
 "Resource": [
 "arn:aws:s3:::my-awesome-bucket"
 ]
 },
 {
 "Effect": "Allow",
 "Action": [
 "s3:PutObject",
 "s3:GetObject",
 "s3:DeleteObject",
 "s3:ListObject"
 ],
 "Resource": [
 "arn:aws:s3:::my-awesome-bucket/*"
 ]
 }
 ]
}

You can learn more about IAM role from this youtube video

After these two settings, the EC2 instance can be created using

rc = ec2.create_instances(ImageId='ami-4fffc834',
                          InstanceType='t2.nano',
                          MinCount=1,
                          MaxCount=1,
                          KeyName='my-key',
                          IamInstanceProfile={'Name': 'my-role'},
                          )

Here my-role is the role I created.

more learning resources

• Boto3 quick hands-on
• Automating AWS With Python and Boto3
• Introduction to aws and boto3
• How to access S3 buckets without access credentials