Amazon Web Services (AWS) is a cloud-based platform that provides a wide range of infrastructure, platform, and software services. It was launched in 2006 and has since become one of the most popular cloud computing platforms in the world, used by individuals, small businesses, and large enterprises alike.

AWS is known for its flexibility, scalability, and cost-effectiveness, allowing businesses to pay only for the services they use and scale up or down as needed. Its reliability and security features also make it a popular choice for businesses that need to store and process sensitive data.

AWS provides a wide range of services, including compute, storage, databases, analytics, machine learning, Internet of Things (IoT), security, and more. It also offers a variety of deployment models, including public, private, and hybrid clouds, as well as edge computing services that allow computing to be performed closer to the source of data.

Here are some of the most common services:

  • Compute services, such as Amazon Elastic Compute Cloud (EC2) and AWS Lambda
  • Storage services, such as Amazon Simple Storage Service (S3) and Elastic Block Store (EBS)
  • Database services, such as Amazon Relational Database Service (RDS) and DynamoDB
  • Networking services, such as Amazon Virtual Private Cloud (VPC) and Elastic Load Balancing (ELB)
  • Management and monitoring services, such as AWS CloudFormation and AWS CloudWatch
  • Security and compliance services, such as AWS Identity and Access Management (IAM) and AWS Key Management Service (KMS)
  • Analytics and machine learning services, such as Amazon SageMaker and Amazon Redshift

Users can interact with these services using

  • SSH (Secure Shell) and AWS CLI (Command Line Interface)
  • boto3 (a Python library used to interact with AWS resources)

Now, lets review these services one by one:

EC2

Amazon Elastic Compute Cloud (EC2) is a web service that provides resizable compute capacity in the cloud. EC2 allows users to launch and manage virtual machines, called instances, in the AWS cloud. With EC2, users can select from a variety of instance types optimized for different use cases, and can scale up or down their compute resources as needed. EC2 also allows users to choose from a range of operating systems, including Amazon Linux, Ubuntu, Windows, and others. EC2 instances can be used to run a wide range of applications, from simple web servers to complex, multi-tier applications.

SSH (Secure Shell) is used to establish a secure, encrypted connection with the EC2 instance, allowing you to remotely log in and execute commands on the server. The basic syntax for the SSH command is

ssh [username]@[EC2 instance public DNS]

You will need to replace [username] with the username you created when setting up your EC2 instance, and [EC2 instance public DNS] with the public DNS address for your instance. Once you have established an SSH connection, you can execute commands on the EC2 instance just as you would on a local machine.

Lambda

AWS Lambda is a serverless compute service provided by Amazon Web Services. With Lambda, you can run your code in response to various events such as changes to data in an Amazon S3 bucket or an update to a DynamoDB table. You upload your code to Lambda, and it takes care of everything required to run and scale your code with high availability. AWS Lambda is an event-driven service, meaning it only runs when an event triggers it.

AWS Lambda console or AWS Command Line Interface (CLI) can be used to create, configure, and deploy your Lambda functions. Following commands allow you to perform common tasks such as creating, updating, invoking, and deleting your Lambda functions using the AWS CLI.

aws lambda create-function: Creates a new Lambda function.
aws lambda update-function-code: Uploads new code to an existing Lambda function.
aws lambda invoke: Invokes a Lambda function.
aws lambda delete-function: Deletes a Lambda function.

Boto3 can also be used to create a new Lambda function. Following code uses the boto3.client() method to create a client object for interacting with AWS Lambda. It then defines the code for the Lambda function, which is stored in a ZIP file. Finally, it uses the lambda_client.create_function() method to create the Lambda function, specifying the function name, runtime, IAM role, handler, and code. The response from this method call contains information about the newly created function.

import boto3

# create a client object to interact with AWS Lambda
lambda_client = boto3.client('lambda')

# define the Lambda function's code
code = {'ZipFile': open('lambda_function.zip', 'rb').read()}

# create the Lambda function
response = lambda_client.create_function(
    FunctionName='my-function',
    Runtime='python3.8',
    Role='arn:aws:iam::123456789012:role/lambda-role',
    Handler='lambda_function.handler',
    Code=code
)

print(response)
S3 (Simple Storage Service)

Amazon S3 (Simple Storage Service) is a cloud storage service offered by Amazon Web Services (AWS). It provides a simple web interface to store and retrieve data from anywhere on the internet. S3 is designed to be highly scalable, durable, and secure, making it a popular choice for data storage, backup and archival, content distribution, and many other use cases. S3 allows you to store and retrieve any amount of data at any time, from anywhere on the web. It also offers different storage classes to help optimize costs based on access frequency and retrieval times.

Following AWS CLI code can be used in SSH to upload, download, and delete a file from S3:

aws s3 cp /path/to/local/file s3://bucket-name/key-name  #code to upload
aws s3 cp s3://bucket-name/key-name /path/to/local/file  #code to download
aws s3 rm s3://bucket-name/key-name  #code to delete a file

Following boto3 code can be used to upload, download, and delete a file from S3. Note that you need to replace your-region, your-access-key, your-secret-key, your-bucket-name, your-file-name, and new-file-name with your own values. Also, make sure that you have the necessary permissions to read, write, and delete objects in your S3 bucket.

import boto3

# set the S3 region and access keys
s3 = boto3.resource('s3', region_name='your-region', aws_access_key_id='your-access-key', aws_secret_access_key='your-secret-key')

# read a file from S3
bucket_name = 'your-bucket-name'
file_name = 'your-file-name'
object = s3.Object(bucket_name, file_name)
file_content = object.get()['Body'].read().decode('utf-8')
print(file_content)

# write a file to S3
new_file_name = 'new-file-name'
new_file_content = 'This is the content of the new file.'
object = s3.Object(bucket_name, new_file_name)
object.put(Body=new_file_content.encode('utf-8'))

# delete a file from S3
object = s3.Object(bucket_name, file_name)
object.delete()

Here’s an example of how to use a KMS key to encrypt and decrypt data in S3 using boto3:

import boto3

# Create a KMS key
kms = boto3.client('kms')
response = kms.create_key()
kms_key_id = response['KeyMetadata']['KeyId']

# Create an S3 bucket or select an existing one
s3 = boto3.resource('s3')
bucket_name = 'my-s3-bucket'

# Grant permissions to the KMS key
key_policy = {
    "Version": "2012-10-17",
    "Id": "key-policy",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::123456789012:root"},
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::123456789012:user/username"},
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
    ]
}
kms.put_key_policy(KeyId=kms_key_id, Policy=json.dumps(key_policy))

# Enable default encryption on the S3 bucket
s3_bucket = s3.Bucket(bucket_name)
s3_bucket.put_encryption(
    ServerSideEncryptionConfiguration={
        'Rules': [
            {
                'ApplyServerSideEncryptionByDefault': {
                    'SSEAlgorithm': 'aws:kms',
                    'KMSMasterKeyID': kms_key_id
                }
            }
        ]
    }
)

# Upload an object to S3 with encryption
s3_client = boto3.client('s3')
s3_client.put_object(
    Bucket=bucket_name,
    Key='example_object',
    Body=b'Hello, world!',
    ServerSideEncryption='aws:kms',
    SSEKMSKeyId=kms_key_id
)

# Download an object from S3 with decryption
response = s3_client.get_object(
    Bucket=bucket_name,
    Key='example_object'
)
body = response['Body'].read()
print(body.decode())

Example code that uses the boto3 library to connect to an S3 bucket, list the contents of a folder, and download the latest file based on its modified timestamp:


import boto3
from datetime import datetime

# set up S3 client
s3 = boto3.client('s3')

# specify bucket and folder
bucket_name = 'my-bucket'
folder_name = 'my-folder/'

# list all files in the folder
response = s3.list_objects(Bucket=bucket_name, Prefix=folder_name)

# sort the files by last modified time
files = response['Contents']
files = sorted(files, key=lambda k: k['LastModified'], reverse=True)

# download the latest file
latest_file = files[0]['Key']
s3.download_file(bucket_name, latest_file, 'local_filename')
EBS (Elastic Block Storage)

Amazon Elastic Block Store (EBS) is a block-level storage service that provides persistent block storage volumes for use with Amazon EC2 instances. EBS volumes are highly available and reliable storage volumes that can be attached to running instances, allowing you to store persistent data separate from the instance itself. EBS volumes are designed for mission-critical systems, so they are optimized for low-latency and consistent performance. EBS also supports point-in-time snapshots, which can be used for backup and disaster recovery.

Using SSH/AWS CLI to manage Elastic Block Store (EBS) volumes

aws ec2 create-volume --availability-zone us-east-1a --size 10 --volume-type gp2

aws ec2 attach-volume --volume-id vol-0123456789abcdef --instance-id i-0123456789abcdef --device /dev/sdf

aws ec2 detach-volume --volume-id vol-0123456789abcdef

aws ec2 delete-volume --volume-id vol-0123456789abcdef

Using python/boto3 to manage Elastic Block Store (EBS) volumes

import boto3

# create an EC2 client object
ec2 = boto3.client('ec2')

# create an EBS volume
response = ec2.create_volume(
    AvailabilityZone='us-west-2a',
    Encrypted=False,
    Size=100,
    VolumeType='gp2'
)
print(response)

# attach a volume
response = ec2.attach_volume(
    Device='/dev/sdf',
    InstanceId='i-0123456789abcdef0',
    VolumeId='vol-0123456789abcdef0'
)
print(response)

# detach a volume
response = ec2.detach_volume(
    VolumeId='vol-0123456789abcdef0'
)
print(response)

# delete a volume
response = ec2.delete_volume(
    VolumeId='vol-0123456789abcdef0'
)
print(response)
RDS (Relational Database Service)

Amazon Relational Database Service (Amazon RDS) is a managed database service offered by Amazon Web Services (AWS) that simplifies the process of setting up, operating, and scaling a relational database in the cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks, freeing up developers to focus on applications and customers. With Amazon RDS, you can choose from several different database engines, including Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server.

To use AWS RDS with PostgreSQL, follow these steps:

  • Log in to your AWS console and navigate to the RDS dashboard.
  • Click the “Create database” button.
  • Choose “Standard Create” and select “PostgreSQL” as the engine.
  • Choose the appropriate version of PostgreSQL that you want to use.
  • Set up the rest of the database settings, such as the instance size, storage, and security group settings.
  • Click the “Create database” button to create your RDS instance.
  • Once the instance is created, you can connect to it using a PostgreSQL client, such as pgAdmin or the psql command-line tool.

To connect to the RDS instance using pgAdmin, follow these steps:

  • Open pgAdmin and right-click on “Servers” in the Object Browser.
  • Click “Create Server”.
  • Enter a name for the server and switch to the “Connection” tab.
  • Enter the following information: – Host: This is the endpoint for your RDS instance, which you can find in the RDS dashboard. – Port: This is the port number for your PostgreSQL instance, which is usually 5432. – Maintenance database: This is the name of the default database that you want to connect to. – Username: This is the username that you specified when you created the RDS instance. – Password: This is the password that you specified when you created the RDS instance.
  • Click “Save” to create the server.

You can now connect to the RDS instance by double-clicking on the server name in the Object Browser.

DynamoDB

Amazon DynamoDB is a fully-managed NoSQL database service that provides fast and predictable performance with seamless scalability. DynamoDB lets users offload the administrative burdens of operating and scaling a distributed database so that they don’t have to worry about hardware provisioning, setup, and configuration, replication, software patching, or cluster scaling. DynamoDB is known for its high performance, ease of use, and flexibility. It supports document and key-value store models, making it suitable for a wide range of use cases, including mobile and web applications, gaming, ad tech, IoT, and more.

Using SSH/AWS CLI to access DynamoDB:

aws dynamodb create-table --table-name <table-name> --attribute-definitions AttributeName=<attribute-name>,AttributeType=S --key-schema AttributeName=<attribute-name>,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

aws dynamodb put-item --table-name <table-name> --item '{"<attribute-name>": {"S": "<attribute-value>"}}'

aws dynamodb get-item --table-name <table-name> --key '{"<attribute-name>": {"S": "<attribute-value>"}}'

aws dynamodb delete-item --table-name <table-name> --key '{"<attribute-name>": {"S": "<attribute-value>"}}'

aws dynamodb delete-table --table-name <table-name>

Using boto3 to work with DynamoDB:

import boto3

# create a DynamoDB resource
dynamodb = boto3.resource('dynamodb')

# create a table
table = dynamodb.create_table(
    TableName='my_table',
    KeySchema=[
        {
            'AttributeName': 'id',
            'KeyType': 'HASH'
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'id',
            'AttributeType': 'S'
        }
    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 5,
        'WriteCapacityUnits': 5
    }
)

# put an item into the table
table.put_item(
    Item={
        'id': '1',
        'name': 'Alice',
        'age': 30
    }
)

# get an item from the table
response = table.get_item(
    Key={
        'id': '1'
    }
)

# print the item
item = response['Item']
print(item)

# delete the table
table.delete()
Amazon Virtual Private Cloud (VPC)

It is a service offered by Amazon Web Services (AWS) that enables users to launch Amazon Web Services resources into a virtual network that they define. With Amazon VPC, users can define a virtual network topology including subnets and routing tables, and control network security using firewall rules and access control lists.

AWS Elastic Load Balancer (ELB)

It is a managed load balancing service provided by Amazon Web Services. It automatically distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers, and IP addresses, in one or more Availability Zones. ELB allows you to easily scale your application by increasing or decreasing the number of resources, such as EC2 instances, behind the load balancer, and it provides high availability and fault tolerance for your applications. There are three types of load balancers available in AWS: Application Load Balancer, Network Load Balancer, and Classic Load Balancer.

AWS CloudFormation

It is a service provided by Amazon Web Services that helps users model and set up their AWS resources. It allows users to create templates of AWS resources in a declarative way, which can then be versioned and managed like any other code. AWS CloudFormation automates the deployment and updates of the AWS resources specified in the templates. This service makes it easier to manage and maintain infrastructure as code and provides a simple way to achieve consistency and repeatability across environments. Users can define the infrastructure for their applications and services, and then AWS CloudFormation takes care of provisioning, updating, and deleting resources, based on the templates defined by the user.

AWS CloudWatch

It is a monitoring service provided by Amazon Web Services (AWS) that collects, processes, and stores log files and metrics from AWS resources and custom applications. With CloudWatch, users can collect and track metrics, collect and monitor log files, and set alarms. CloudWatch is designed to help users identify and troubleshoot issues, and it can be used to monitor AWS resources such as EC2 instances, RDS instances, and load balancers, as well as custom metrics and logs from any other application running on AWS. CloudWatch can also be used to gain insights into the performance and health of applications and infrastructure, and it integrates with other AWS services such as AWS Lambda, AWS Elastic Beanstalk, and AWS EC2 Container Service.

AWS Key Management Service (KMS)

It is a managed service that makes it easy for you to create and control the encryption keys used to encrypt your data. KMS is integrated with many other AWS services, such as Amazon S3, Amazon EBS, and Amazon RDS, allowing you to easily protect your data with encryption. With KMS, you can create, manage, and revoke encryption keys, and you can audit key usage to ensure compliance with security best practices. KMS also supports hardware security modules (HSMs) for added security.

import boto3

# create a KMS client
client = boto3.client('kms')

# Encrypt data using a KMS key
response = client.encrypt(
    KeyId='alias/my-key', 
    Plaintext=b'My secret data'
)

# Decrypt data using a KMS key
response = client.decrypt(
    CiphertextBlob=response['CiphertextBlob']
)

# Manage encryption keys
# Create a new KMS key
response = client.create_key(
    Description='My encryption key',
    KeyUsage='ENCRYPT_DECRYPT',
    Origin='AWS_KMS'
)

# List all KMS keys in your account
response = client.list_keys()

# Handle errors gracefully when using KMS
import botocore.exceptions
try:
    response = client.encrypt(
        KeyId='alias/my-key', 
        Plaintext=b'My secret data'
    )
except botocore.exceptions.ClientError as error:
    print(f'An error occurred: {error}')
Amazon SageMaker

It is a fully-managed service that enables developers and data scientists to easily build, train, and deploy machine learning models at scale. With SageMaker, you can quickly create an end-to-end machine learning workflow that includes data preparation, model training, and deployment. The service offers a variety of built-in algorithms and frameworks, as well as the ability to bring your own custom algorithms and models.

SageMaker provides a range of tools and features to help you manage your machine learning projects. You can use the built-in Jupyter notebooks to explore and visualize your data, and use SageMaker’s automatic model tuning capabilities to find the best hyperparameters for your model. The service also offers integration with other AWS services such as S3, IAM, and CloudWatch, making it easy to build and deploy machine learning models in the cloud.

With SageMaker, you only pay for what you use, with no upfront costs or long-term commitments. The service is designed to scale with your needs, so you can start small and grow your machine learning projects as your data and business needs evolve.

Amazon Redshift

It is a cloud-based data warehousing service provided by Amazon Web Services (AWS). It is a fully managed, petabyte-scale data warehouse that enables businesses to analyze data using existing SQL-based business intelligence tools. Redshift is designed to handle large data sets and to scale up or down as needed, making it a flexible and cost-effective solution for data warehousing.

Using redshift-data to connect to Redshift cluster:

import boto3

# Connect to the Redshift cluster
client = boto3.client('redshift-data', region_name='us-west-2')
response = client.connect(Database='my_database', DbUser='my_user', DbPassword='my_password', ClusterIdentifier='my_cluster')

# Execute a SQL statement
response = client.execute_statement(Sql='SELECT * FROM my_table', ConnectionId=response['ConnectionId'])

# Fetch the results
statement_id = response['Id']
response = client.get_statement_result(Id=statement_id)

# Print the results
for row in response['Records']:
    print(row)

# parse the response JSON and create a Pandas DataFrame
# we parse the JSON using json.dumps and create a Pandas DataFrame from the resulting string using pd.read_json. The resulting DataFrame will have the columns and rows returned by the SQL query.
df = pd.read_json(json.dumps(response['Records']))

Using psycopg2 to connect to Redshift cluster:

pip install boto3 psycopg2

import boto3
client = boto3.client('redshift')

# create a new Redshift cluster using boto3:
response = client.create_cluster(
    ClusterIdentifier='my-redshift-cluster',
    NodeType='dc2.large',
    MasterUsername='myusername',
    MasterUserPassword='mypassword',
    ClusterSubnetGroupName='my-subnet-group',
    VpcSecurityGroupIds=['my-security-group'],
    ClusterParameterGroupName='default.redshift-1.0',
    NumberOfNodes=2,
    PubliclyAccessible=False,
    Encrypted=True,
    HsmClientCertificateIdentifier='my-hsm-certificate',
    HsmConfigurationIdentifier='my-hsm-config',
    Tags=[{'Key': 'Name', 'Value': 'My Redshift Cluster'}]
)
print(response)

# connect to the Redshift cluster and create a new database
import psycopg2
conn = psycopg2.connect(
    host='my-redshift-cluster.xxxxxxxx.us-west-2.redshift.amazonaws.com',
    port=5439,
    dbname='mydatabase',
    user='myusername',
    password='mypassword'
)
cur = conn.cursor()
cur.execute('CREATE DATABASE mynewdatabase')
conn.commit()
cur.close()
conn.close()

# Query the Redshift database
import psycopg2
conn = psycopg2.connect(
    host='my-redshift-cluster.xxxxxxxx.us-west-2.redshift.amazonaws.com',
    port=5439,
    dbname='mynewdatabase',
    user='myusername',
    password='mypassword'
)
cur = conn.cursor()
cur.execute('SELECT * FROM mytable')
for row in cur:
    print(row)
cur.close()
conn.close()

# Delete the Redshift cluster using boto3
import boto3
client = boto3.client('redshift')
response = client.delete_cluster(
    ClusterIdentifier='my-redshift-cluster',
    SkipFinalClusterSnapshot=True
)
print(response)

Comments welcome!