AWS S3: Complete Guide to Cloud Object Storage

Step 1: Set Up AWS IAM User for S3 Access

Before interacting with S3, you need an IAM user with the necessary permissions. This ensures you operate with restricted privileges, not as the root account.

1. Log in to the AWS Management Console: Go to console.aws.amazon.com.
2. Navigate to IAM: Search for "IAM" in the search bar and select it.
3. Create a New User:
   • In the IAM dashboard, click `Users` in the left navigation pane.
   • Click `Add users`.
   • Enter a `User name` (e.g., `s3-admin-user`).
   • For `AWS credential type`, select `Access key - Programmatic access`. Click `Next`.
4. Attach Permissions:
   • On the `Set permissions` page, select `Attach policies directly`.
   • Search for `AmazonS3FullAccess` and select the checkbox. (For production, narrow this down to specific bucket/actions).
   • Click `Next`.
5. Review and Create:
   • Review the user details and permissions.
   • Click `Create user`.
6. Save Credentials:
   • On the `Retrieve credentials` page, you will see the `Access key ID` and `Secret access key`.
   • Crucially, download the .csv file or copy these keys immediately. This is your ONLY chance to view the Secret access key.
   • Store these credentials in a secure location. You'll need them for `s3cmd` configuration.

Step 2: Create an S3 Bucket (via AWS Console)

Buckets are the fundamental containers for your objects in S3. They are globally unique.

1. Navigate to S3: From the AWS Console, search for "S3" and select it.
2. Create Bucket: Click the `Create bucket` button.
3. Configure Bucket Properties:
   • AWS Region: Choose a region closest to your users/services (e.g., `us-east-1`, `eu-central-1`).
   • Bucket name: Enter a globally unique, DNS-compliant name (e.g., `my-unique-pocketcursor-bucket-2023`). **Avoid sensitive data in the name.**
   • Object Ownership: For new accounts, AWS recommends `ACLs disabled (recommended)`. Keep this default unless you have specific legacy use cases.
   • Block Public Access settings for this bucket: **Highly Recommended to keep all four options CHECKED.** This prevents accidental public exposure of your data. You can adjust this later if you intend to host a static website (Step 5).
   • Bucket Versioning: (Optional but recommended for data protection) Enable this to keep multiple versions of an object, protecting against accidental deletions or overwrites.
   • Tags: (Optional) Add tags for cost allocation or organization.
   • Default encryption: (Optional but recommended) Enable `Server-side encryption with Amazon S3 managed keys (SSE-S3)` for data at rest.
4. Create Bucket: Click `Create bucket` at the bottom.

Step 3: Understanding S3 Storage Classes

S3 offers various storage classes, each designed for specific access patterns and pricing models. Choosing the right class helps optimize costs significantly.

• S3 Standard:
  • Use Case: Frequently accessed data (e.g., dynamic websites, content distribution, mobile applications).
  • Availability/Durability: High, spread across at least three Availability Zones (AZs).
  • Retrieval: Millisecond access.
  • Cost: Higher storage cost, low retrieval cost.
• S3 Intelligent-Tiering:
  • Use Case: Data with unknown or changing access patterns.
  • Availability/Durability: High, automatically moves data between access tiers (Frequent, Infrequent, Archive Instant Access, Archive Deep Access) based on usage.
  • Cost: Storage cost varies, plus a small monthly monitoring and automation fee per object.
• S3 Standard-Infrequent Access (S3 Standard-IA):
  • Use Case: Long-lived, less frequently accessed data that needs rapid access when required (e.g., backups, disaster recovery, older analytics data).
  • Availability/Durability: High, spread across at least three AZs.
  • Retrieval: Millisecond access, but incurs a retrieval fee.
  • Cost: Lower storage cost than Standard, higher retrieval cost.
• S3 One Zone-Infrequent Access (S3 One Zone-IA):
  • Use Case: Same as S3 Standard-IA, but for non-critical, reproducible data that can tolerate losing data in an AZ disaster (e.g., secondary backups, easily recreated media archives).
  • Availability/Durability: High durability, but stored in a single AZ, so data loss is possible if that AZ is destroyed.
  • Retrieval: Millisecond access, incurs retrieval fee.
  • Cost: Lower storage cost than S3 Standard-IA.
• Amazon S3 Glacier Instant Retrieval:
  • Use Case: Long-term archive data that needs immediate access (e.g., medical images, news media assets).
  • Availability/Durability: High, spread across multiple AZs.
  • Retrieval: Millisecond access, higher retrieval cost than IA.
  • Cost: Very low storage cost.
• Amazon S3 Glacier Flexible Retrieval (formerly S3 Glacier):
  • Use Case: Archival data where retrieval times of minutes to hours are acceptable (e.g., backups, disaster recovery).
  • Availability/Durability: High, spread across multiple AZs.
  • Retrieval: Configurable from minutes (expedited) to 5-12 hours (standard).
  • Cost: Extremely low storage cost.
• Amazon S3 Glacier Deep Archive:
  • Use Case: Long-term data archiving, lowest cost, for data accessed rarely (e.g., regulatory compliance archives, historical research data).
  • Availability/Durability: High, spread across multiple AZs.
  • Retrieval: Configurable from 12 hours (standard) to 48 hours (bulk).
  • Cost: Lowest storage cost of all S3 classes.

Step 4: Managing Objects (Upload, Download, Delete via AWS Console)

Once you have a bucket, you can start storing objects. Let's cover basic object management using the AWS Console.

1. Navigate to your Bucket: From the S3 dashboard, click on the name of the bucket you created.
2. Upload an Object:
   • Click the `Upload` button.
   • Drag and drop files from your computer, or click `Add files`.
   • (Optional) You can set specific storage class, encryption, or metadata for each object during upload.
   • Click `Upload`.
3. Create a Folder (Prefix):
   • S3 doesn't have true folders, but uses "prefixes" (like `images/cat.jpg`). You can create a "folder" by clicking `Create folder` and naming it (e.g., `my-photos/`).
4. Download an Object:
   • Select the object(s) you want to download by checking the box next to their name.
   • Click the `Download` button.
5. Delete an Object:
   • Select the object(s) you want to delete.
   • Click the `Delete` button.
   • Confirm the deletion by typing `permanently delete` in the confirmation box and clicking `Delete objects`.
6. View Object Details:
   • Click on an object's name (not the checkbox) to view its properties: URL, ETag, size, storage class, encryption status, metadata, and permissions.

Step 5: Setting Permissions (IAM Policies & Bucket Policies)

Controlling who can access your S3 buckets and objects is paramount. S3 offers several layers of permissions, with IAM policies and Bucket policies being the most common and powerful.

A. Identity-Based Policies (IAM Policies)

These are attached to IAM users, groups, or roles and define what actions that identity can perform on S3 resources.

Example: Read-only access to a specific bucket for an IAM user:

When creating an IAM user (Step 1), instead of `AmazonS3FullAccess`, you would attach a custom policy like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

B. Bucket Policies

These are JSON policies directly attached to an S3 bucket. They can grant or deny access to specific AWS accounts, IAM users, or even anonymous (public) users. Bucket policies are often used for cross-account access or to make a bucket public for static website hosting.

Example: Public Read-Only Access (for Static Website Hosting)

To host a static website, you'll need to allow public read access. **This requires disabling "Block Public Access" for the bucket first (Step 2).**

1. Disable Block Public Access:
   • Go to your bucket in the S3 console.
   • Click the `Permissions` tab.
   • Under `Block public access (bucket settings)`, click `Edit`.
   • **Uncheck** `Block all public access`.
   • **Uncheck** `Block public access to buckets and objects granted through new public ACLs`. (Keep the other two checked if you only want bucket policies to grant public access).
   • Click `Save changes` and type `confirm` to proceed.
2. Apply Bucket Policy:
   • Still on the `Permissions` tab, scroll down to `Bucket policy`. Click `Edit`.
   • Paste the following policy, replacing `your-bucket-name` with your actual bucket name:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

3. Click `Save changes`. Your bucket content (e.g., `index.html`) can now be publicly read.

C. Access Control Lists (ACLs)

ACLs are a legacy access control mechanism that predates IAM policies. They grant basic read/write permissions to specific AWS accounts or predefined groups. AWS generally recommends using IAM policies and bucket policies over ACLs for most use cases, and for new buckets, ACLs are often disabled by default. We won't delve into detailed ACL management here but acknowledge their existence.

Step 6: Accessing S3 with `s3cmd` (Command-Line Tool)

`s3cmd` is a free, open-source command-line tool for managing S3 buckets and objects. It's an excellent way to automate tasks and interact with S3 directly from your Linux server.

A. Install `s3cmd` on Ubuntu

B. Configure `s3cmd`

You'll need the `Access key ID` and `Secret access key` from Step 1.

Follow the prompts:

• `Access Key:` Paste your `Access key ID`.
• `Secret Key:` Paste your `Secret access key`.
• `Default Region:` Enter the region where your bucket is (e.g., `us-east-1`, `eu-central-1`).
• `Encryption password:` (Optional, for client-side encryption) You can leave this blank if you're using server-side encryption or don't need this feature.
• `Path to GPG program:` (Optional) Leave blank if not using GPG encryption.
• `Use HTTPS protocol:` `Yes` (recommended for security).
• `Use 's3.amazonaws.com' as endpoint:` `No` (unless using `us-east-1` directly, let `s3cmd` determine by region).
• `HTTP Proxy server name:` (Optional) Leave blank if no proxy.
• **`Test access with supplied credentials?`**: Type `Y`. If successful, you'll see `Success. Your access key and secret key worked fine :)`.

C. Common `s3cmd` Commands

Replace `your-bucket-name` with your actual bucket name, and `local-file.txt` / `remote-path/` with your specific paths.

1. List Buckets:

2. List Objects in a Bucket:

3. Create a Bucket:

4. Upload a File:

5. Download a File:

6. Delete an Object:

7. Sync a Local Directory to S3:

8. Delete a Folder (Recursively):

9. Set Object ACL (e.g., Make Public Read):

10. Change Storage Class of an Object:

11. Get Info on a Bucket or Object:

12. Get Help: