Cloud Storage: Buckets, Objects & Access Control

Google Cloud Storage is a globally unified object storage service for structured and unstructured data. Objects are stored in buckets. There is no directory hierarchy — object names can contain slashes to simulate folders.

Storage Classes

Standard    — frequent access, lowest latency. No minimum duration.
Nearline    — once/month access. 30-day minimum. ~50% cheaper than Standard.
Coldline    — once/quarter access. 90-day minimum. ~75% cheaper.
Archive     — once/year access. 365-day minimum. ~94% cheaper.

Regional vs Multi-regional vs Dual-region:
  Regional        — single region (us-central1). Lower latency within region.
  Multi-region    — spans a continent (US, EU, ASIA). Higher availability.
  Dual-region     — two specific regions. 99.95% SLA, turbo replication option.

Choose class at bucket level or per-object. Lifecycle rules can auto-transition objects.

Creating Buckets & Uploading

# Create bucket
gcloud storage buckets create gs://my-bucket   --location=us-central1   --storage-class=STANDARD

# Create bucket with uniform bucket-level access
gcloud storage buckets create gs://my-bucket   --location=us-central1   --uniform-bucket-level-access

# Upload files
gcloud storage cp local-file.txt gs://my-bucket/
gcloud storage cp -r ./local-dir/ gs://my-bucket/prefix/  # recursive

# Download
gcloud storage cp gs://my-bucket/file.txt ./local/
gcloud storage cp -r gs://my-bucket/prefix/ ./local/

# List objects
gcloud storage ls gs://my-bucket/
gcloud storage ls -l gs://my-bucket/   # with size/date
gcloud storage ls -r gs://my-bucket/  # recursive

# Delete
gcloud storage rm gs://my-bucket/file.txt
gcloud storage rm -r gs://my-bucket/prefix/  # recursive
gcloud storage rm --all-versions gs://my-bucket/file.txt

Access Control

Cloud Storage supports two access control models: fine-grained (legacy ACLs per object) and uniform bucket-level access (IAM-only, recommended).

# Uniform bucket-level access (recommended — IAM controls everything)
gcloud storage buckets update gs://my-bucket   --uniform-bucket-level-access

# Grant IAM roles on bucket
gcloud storage buckets add-iam-policy-binding gs://my-bucket   --member="user:alice@example.com"   --role="roles/storage.objectViewer"

gcloud storage buckets add-iam-policy-binding gs://my-bucket   --member="serviceAccount:my-sa@project.iam.gserviceaccount.com"   --role="roles/storage.objectAdmin"

# Common Storage IAM roles:
# roles/storage.objectViewer     — read objects
# roles/storage.objectCreator    — create/overwrite objects
# roles/storage.objectAdmin      — full object control
# roles/storage.admin            — full bucket + object control

# Make bucket public (all users can read)
gcloud storage buckets add-iam-policy-binding gs://my-bucket   --member="allUsers"   --role="roles/storage.objectViewer"

# Check effective IAM policy
gcloud storage buckets get-iam-policy gs://my-bucket

Python Client Library

from google.cloud import storage

client = storage.Client()

# Create bucket
bucket = client.create_bucket("my-bucket", location="us-central1")

# Upload from file
bucket = client.bucket("my-bucket")
blob = bucket.blob("path/to/object.txt")
blob.upload_from_filename("local-file.txt")
blob.upload_from_string("Hello, World!", content_type="text/plain")

# Download
blob.download_to_filename("local-copy.txt")
content = blob.download_as_text()

# List objects
blobs = client.list_blobs("my-bucket", prefix="images/")
for blob in blobs:
    print(blob.name, blob.size)

# Delete
blob.delete()

# Copy between buckets
source = client.bucket("source-bucket").blob("file.txt")
dest = client.bucket("dest-bucket")
source_bucket = client.bucket("source-bucket")
source_bucket.copy_blob(source, dest, "new-name.txt")

Cloud Storage: gsutil CLI, Lifecycle & Signed URLs

gsutil (Legacy CLI)

gsutil is the older Cloud Storage CLI, still widely used. The newer gcloud storage commands are faster for large transfers.

# Sync (like rsync — only uploads changed files)
gsutil -m rsync -r ./local-dir/ gs://my-bucket/prefix/
gsutil -m rsync -r -d gs://my-bucket/prefix/ ./local/  # delete files not in source

# -m flag: parallel multi-threading (much faster for many files)
gsutil -m cp -r ./images/ gs://my-bucket/images/

# Set metadata
gsutil setmeta -h "Cache-Control:public, max-age=3600" gs://my-bucket/**.js
gsutil setmeta -h "Content-Type:image/webp" gs://my-bucket/image.webp

# Set CORS
# cors.json: [{"origin": ["*"], "method": ["GET"], "maxAgeSeconds": 3600}]
gsutil cors set cors.json gs://my-bucket

# Get bucket info
gsutil ls -L -b gs://my-bucket

# Move/rename
gsutil mv gs://my-bucket/old-name.txt gs://my-bucket/new-name.txt

Lifecycle Rules

Lifecycle rules automatically transition or delete objects based on age, storage class, or other conditions.

{
  "rule": [
    {
      "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
      "condition": {"age": 30}
    },
    {
      "action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
      "condition": {"age": 90}
    },
    {
      "action": {"type": "Delete"},
      "condition": {"age": 365}
    },
    {
      "action": {"type": "Delete"},
      "condition": {
        "numNewerVersions": 3,
        "isLive": false
      }
    }
  ]
}

# Apply lifecycle policy
gcloud storage buckets update gs://my-bucket   --lifecycle-file=lifecycle.json

# View current lifecycle
gcloud storage buckets describe gs://my-bucket --format="yaml(lifecycle)"

# Enable object versioning
gcloud storage buckets update gs://my-bucket --versioning

# List all versions
gcloud storage ls -a gs://my-bucket/

Signed URLs

Signed URLs grant temporary access to a private object without requiring a Google account. Use for letting users download/upload directly from/to Cloud Storage.

# Generate signed URL (requires service account key or signing authority)
gcloud storage sign-url gs://my-bucket/private-file.pdf   --duration=1h   --private-key-file=service-account-key.json

# Or using Python client
from google.cloud import storage
import datetime

client = storage.Client()
bucket = client.bucket("my-bucket")
blob = bucket.blob("private-file.pdf")

url = blob.generate_signed_url(
    expiration=datetime.timedelta(hours=1),
    method="GET",
    version="v4",
)
print(url)  # share this URL — valid for 1 hour

# Signed URL for upload (PUT)
upload_url = blob.generate_signed_url(
    expiration=datetime.timedelta(minutes=15),
    method="PUT",
    content_type="application/pdf",
    version="v4",
)
# Client uploads directly: PUT upload_url with file body

Best Practices

• Use uniform bucket-level access — simpler IAM management, no per-object ACL complexity.

• Enable Object Versioning for critical data — protects against accidental deletes and overwrites.

• Use lifecycle rules to auto-archive aging data to Nearline/Coldline/Archive classes.

• Prefer gcloud storage over gsutil for new scripts — parallel transfers, better performance.

• For public static websites: enable website configuration (MainPageSuffix + NotFoundPage), add allUsers viewer.

• Use Pub/Sub notifications on bucket changes to trigger Cloud Functions for event-driven processing.

• Customer-managed encryption keys (CMEK): use Cloud KMS to control encryption keys for compliance.

• Transfer Service: move large datasets from S3, Azure Blob, or HTTP/HTTPS sources efficiently.