AWS S3: Fundamentals & Bucket Operations

Amazon S3 (Simple Storage Service) is object storage built for any amount of data. Objects are stored in buckets. Each object has a key (path-like name), data, and metadata. S3 is not a filesystem — there are no real directories, only key prefixes.

Core Concepts

• Bucket: globally unique container; region-specific but namespace is global

• Object: key + data + metadata; max object size 5TB (multipart for >100MB)

• Key: full "path" like images/2024/photo.jpg — the slash is just part of the name

• S3 URI format: s3://my-bucket/images/photo.jpg

• Strong consistency: all operations (GET/PUT/DELETE) are strongly consistent since Dec 2020

AWS CLI — Bucket & Object Operations

# Create a bucket (region required outside us-east-1)
aws s3 mb s3://my-bucket --region eu-west-1

# List buckets / objects
aws s3 ls
aws s3 ls s3://my-bucket/
aws s3 ls s3://my-bucket/images/ --recursive --human-readable

# Copy files
aws s3 cp file.txt s3://my-bucket/uploads/file.txt
aws s3 cp s3://my-bucket/file.txt ./local-file.txt
aws s3 cp s3://src-bucket/ s3://dst-bucket/ --recursive

# Sync local dir to S3 (only changed files)
aws s3 sync ./dist s3://my-bucket/website/ --delete
aws s3 sync s3://my-bucket/backups/ ./backups/

# Move and delete
aws s3 mv s3://my-bucket/old.txt s3://my-bucket/new.txt
aws s3 rm s3://my-bucket/file.txt
aws s3 rm s3://my-bucket/folder/ --recursive

Object Metadata & Content-Type

# Upload with metadata and content-type
aws s3 cp index.html s3://my-bucket/ \
  --content-type "text/html; charset=utf-8" \
  --cache-control "max-age=31536000" \
  --metadata '{"x-app-version":"1.2.0"}'

# Set content-type on existing objects (requires copy to itself)
aws s3 cp s3://my-bucket/file.js s3://my-bucket/file.js \
  --metadata-directive REPLACE \
  --content-type "application/javascript" \
  --cache-control "max-age=86400"

Presigned URLs

# Generate a presigned GET URL (expires in 1 hour)
aws s3 presign s3://my-bucket/private/report.pdf --expires-in 3600

# Presigned PUT URL via SDK (CLI only supports GET)
# Use aws s3api for more control

// Node.js SDK v3
import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const client = new S3Client({ region: 'eu-west-1' });

// Presigned GET — share private file for 1 hour
const getUrl = await getSignedUrl(client, new GetObjectCommand({
  Bucket: 'my-bucket',
  Key: 'private/report.pdf',
}), { expiresIn: 3600 });

// Presigned PUT — allow client to upload directly to S3
const putUrl = await getSignedUrl(client, new PutObjectCommand({
  Bucket: 'my-bucket',
  Key: `uploads/${crypto.randomUUID()}.jpg`,
  ContentType: 'image/jpeg',
}), { expiresIn: 300 });

Multipart Upload (large files)

# aws s3 cp handles multipart automatically for files > 8MB
# For fine-grained control use s3api:

# 1. Initiate
aws s3api create-multipart-upload --bucket my-bucket --key large-file.zip

# 2. Upload parts (each must be >= 5MB except last)
aws s3api upload-part --bucket my-bucket --key large-file.zip \
  --part-number 1 --upload-id <UploadId> --body part1.bin

# 3. Complete
aws s3api complete-multipart-upload --bucket my-bucket --key large-file.zip \
  --upload-id <UploadId> \
  --multipart-upload '{"Parts":[{"ETag":"...","PartNumber":1}]}'

# List and abort stuck uploads (important — they cost money)
aws s3api list-multipart-uploads --bucket my-bucket
aws s3api abort-multipart-upload --bucket my-bucket --key large-file.zip --upload-id <UploadId>