Security

Secrets Management: Store, Rotate, Never Leak

Where API keys and passwords really leak, why env vars deliver secrets but do not protect them, and how secret managers and short-lived credentials fix it.

Intermediate·14 min read·Updated Oct 6, 2026

A secret is any string that grants access by itself: whoever reads it is you, as far as the other system can tell. So the job is not hiding the string well once; it is keeping it out of every place that gets copied (git history, image layers, logs, CI output, browser bundles), storing it where access is controlled and audited, and making it short-lived so that a leak expires on its own. The strongest setups have no long-lived secret at all: workloads prove their identity and receive credentials that last minutes.

Context

Early web apps kept database passwords in a config file next to the code, and the config file usually ended up in the repository. The Twelve-Factor App guidelines (2011) moved configuration into environment variables so the same build could run anywhere, and .env files became the local version of that. As services multiplied, dedicated stores appeared: HashiCorp Vault (2015), AWS Secrets Manager (2018), Google Cloud Secret Manager and others. The next step removed stored credentials altogether: cloud IAM (identity and access management) roles for servers, and since late 2021 OIDC (OpenID Connect) federation that lets a CI (continuous integration) job such as GitHub Actions log in to AWS without any saved key.

The incidents are the reason it matters. In 2016 attackers found AWS keys in a private Uber GitHub repository and used them to download data on 57 million users; in 2022 Toyota disclosed that an access key had sat in a public repository for almost five years. You have handled secrets every time you pasted an API key into a .env file:

payments.ts

// .env (git-ignored):  STRIPE_SECRET_KEY=sk_live_51Hx...
import Stripe from 'stripe'

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!)
// anyone holding this one string can charge, refund and export customers

Secret: A value that authenticates on its own: API keys, database passwords, signing keys, OAuth client secrets, private keys, webhook secrets.
Secret manager: A service that stores secrets encrypted, hands them out only to authorized identities, and logs every read. Vault, AWS Secrets Manager, GCP Secret Manager, Doppler, 1Password.
Workload identity: Proving who a server, pod or CI job is (via its cloud role or a signed token) instead of giving it a stored password.
Rotation: Replacing a secret with a new value and revoking the old one, ideally on a schedule and without downtime.
KMS (key management service): A service that holds encryption keys in hardware and performs encrypt/decrypt operations, so the master key itself never leaves it.

Why it matters

Leaked credentials are among the most common ways into a company, because using one needs no exploit: the attacker simply logs in. Bots scan every public GitHub push and try fresh cloud keys, often within minutes, and GitGuardian counted more than 12 million new secrets in public GitHub commits in 2023 alone. Inside a company the damage spreads further, because one key copied into a dozen places can no longer be revoked without breaking a dozen things, so nobody rotates it.

Where secrets actually leak

Secrets rarely leak from the place they are stored. They leak from the places they get copied to, and most of those copies are permanent: git keeps every committed version, an image layer keeps every file that was ever in it, and a log line is shipped to three systems before anyone notices.

git

.env committed oncedeleted next commit, still in historyforks and clones keep it too

image

COPY .env into the imagebuild ARG in layer historyanyone who can pull can read

browser

NEXT_PUBLIC_ / VITE_ variableinlined into the JS bundlepublic by design

runtime

config object logged at startuperror page or crash report with envshipped to log storage

echo $TOKEN in a stepsecrets exposed to fork PR jobsbuild logs are often public

Every red chip is a copy that outlives the moment it was made. Deleting the file later does not remove any of them.

The browser case trips up full-stack frameworks in particular. Next.js inlines every NEXT_PUBLIC_ variable into the client JavaScript at build time, and Vite does the same with VITE_, so adding the prefix to "make it work in the component" publishes the value. Docker has the same trap at build time: a file deleted in a later step still exists in the earlier layer.

Dockerfile

# bad: the file stays in this layer even if a later RUN deletes it
COPY .env /app/.env

# good (BuildKit): mounted only during this RUN, never written to a layer
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm ci

# build: docker build --secret id=npm_token,env=NPM_TOKEN .

Environment variables vs secret managers

Environment variables are a fine way to deliver a secret to a process and a poor place to keep one. Every child process inherits them, so a shell-out to git or ImageMagick gets your database password too. On Linux they are readable from /proc/<pid>/environ by the same user, they end up in crash dumps and in error reporters that capture the environment, and the value has no access log, no expiry and no owner. A secret manager adds what is missing: encryption at rest, per-identity access control, an audit trail of every read, versioning and rotation.

Where it lives	Encrypted at rest	Who can read it	Rotation
.env in the repo	No	Anyone with a clone, forever	Manual, and the old value stays in history
Platform env vars	Yes, by the provider	Project members, the process and its children	Manual edit plus redeploy
Kubernetes Secret	Only if configured; base64 by default	Anyone allowed to read Secrets in the namespace	Manual, pods may need a restart
Secret manager	Yes	Named identities, every read audited	Built in, or dynamic per-client secrets
Workload identity	Nothing stored	Whatever the role’s trust policy allows	Automatic, credentials last minutes to hours

Dynamic secrets go one step further. Instead of storing a database password, Vault creates a fresh database user for each application instance when it asks, with a TTL (time to live) of, say, one hour, and drops it when the lease expires. A leaked credential then works for at most an hour and points at exactly one instance.

Short-lived credentials instead of stored keys

The safest secret is the one that does not exist. Cloud platforms let a workload prove what it is and exchange that proof for temporary credentials: an EC2 instance or a Kubernetes service account gets an IAM role, and a GitHub Actions job gets a signed OIDC token from GitHub that AWS STS (Security Token Service) accepts in place of an access key. Nothing long-lived is saved anywhere, so there is nothing to leak or rotate.

OIDC federation from CI to AWS. The job never holds a stored key: GitHub signs a token describing the job, AWS checks it against the role's trust policy and returns credentials that expire within the hour.

deploy.yml

permissions:
  id-token: write      # let the job request an OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-deploy
          aws-region: eu-central-1
      # no AWS_ACCESS_KEY_ID anywhere; these credentials expire
      - run: aws s3 sync ./dist s3://my-site

The security of this setup lives in the role's trust policy. It must pin the token's subject to your repository and branch; a policy that only checks the audience lets any GitHub repository assume the role.

trust-policy.json

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub":
      "repo:acme/site:ref:refs/heads/main"
  }
}

Rotation, envelope encryption and leak response

Rotating without downtime

A rotation that swaps one value for another breaks every instance that still holds the old one. Zero-downtime rotation keeps two valid credentials for a while. AWS Secrets Manager models exactly this with its AWSPENDING and AWSCURRENT stages, and Vault does it by alternating between two database users.

1
Create credential B alongside the current credential A. Both are valid.
2
Store B as the new current version in the secret manager. Applications that fetch on start-up or on a timer pick it up; long-running ones need a refresh or a rolling restart.
3
Confirm nothing uses A anymore: the provider's "last used" timestamp, database connection logs, or CloudTrail.
4
Revoke A. If something breaks now, it was still holding A, and B is already in place to switch to.

Envelope encryption

When the application must encrypt data itself, for example API tokens of your own customers, it should not hold a long-lived encryption key either. With envelope encryption, KMS generates a fresh DEK (data encryption key) for each record or file, returns it both in plaintext and encrypted under a KEK (key encryption key) that never leaves KMS. You encrypt the data locally with the plaintext DEK, store the encrypted DEK next to the ciphertext, and throw the plaintext DEK away. Rotating the KEK, revoking access or auditing reads all happen in KMS.

envelope.ts

import {KMSClient, GenerateDataKeyCommand, DecryptCommand}
  from '@aws-sdk/client-kms'
import {createCipheriv, createDecipheriv, randomBytes} from 'node:crypto'

const kms = new KMSClient({})
const KEY_ID = 'alias/app-data'          // the KEK never leaves KMS

export async function encrypt(plaintext: Buffer) {
  const {Plaintext: dek, CiphertextBlob: wrappedKey} = await kms.send(
    new GenerateDataKeyCommand({KeyId: KEY_ID, KeySpec: 'AES_256'}))
  const iv = randomBytes(12)
  const c = createCipheriv('aes-256-gcm', dek!, iv)
  const data = Buffer.concat([c.update(plaintext), c.final()])
  // store wrappedKey next to the data; the plaintext dek is discarded
  return {data, iv, tag: c.getAuthTag(), wrappedKey: wrappedKey!}
}

export async function decrypt(e: Awaited<ReturnType<typeof encrypt>>) {
  const {Plaintext: dek} = await kms.send(
    new DecryptCommand({CiphertextBlob: e.wrappedKey}))
  const d = createDecipheriv('aes-256-gcm', dek!, e.iv)
  d.setAuthTag(e.tag)
  return Buffer.concat([d.update(e.data), d.final()])
}

The cipher here is AES (Advanced Encryption Standard) in GCM (Galois/Counter Mode), which also authenticates the data, so a tampered ciphertext fails to decrypt instead of returning garbage.

When a secret leaks

1
Revoke or rotate first. Assume it was copied the moment it became visible; the clean-up comes after.
2
Check what the key did during the exposure window: CloudTrail, Stripe or GitHub audit logs, database connection logs.
3
Remove it from the code. Rewriting git history is optional hygiene, not a fix: forks, clones, CI caches and pull-request refs keep the old commits.
4
Close the path it leaked through: scanning in pre-commit (gitleaks), on push (GitHub push protection, on by default for public repositories since early 2024) and in CI (trufflehog), plus moving that secret to short-lived credentials if the platform allows.

Pitfalls

Deleting the commit instead of rotating the key
A force-push hides the commit in your branch view but not from the clones, forks and scrapers that already fetched it, and GitHub can keep the object reachable through pull-request refs. The only thing that ends the exposure is making the leaked value useless, so rotation comes first, always.
A public prefix on a server secret
NEXT_PUBLIC_STRIPE_SECRET_KEY works in the component because the build pastes the value into the bundle that every visitor downloads. Secrets belong in server code: route handlers, server actions, or a backend endpoint the client calls.
Logging configuration or the whole environment
A startup line like console.log(config), a debug error page, or a crash reporter that attaches process.env copies every secret into log storage, which has wider access and longer retention than the secret manager. Redact by key name in the logger and never serialize the full environment.
One long-lived key shared everywhere
The same API key in development, staging, production and three CI pipelines means a leak anywhere is a production incident, and rotating it requires touching every place at once, so it never happens. Give each environment and workload its own credential with the narrowest permissions that work.
Assuming Kubernetes Secrets are encrypted
Base64 is encoding. Without encryption at rest, anyone with access to etcd or its backups reads every secret, and RBAC that allows listing Secrets is effectively read access to all of them. Enable a KMS encryption provider or sync from an external secret manager.

Interview questions

Q1Why are environment variables not enough to protect secrets?

They deliver a secret to a process but give it no protection after that. Every child process inherits them, they can be read from the process environment, and they end up in crash dumps, debug pages and logs. There is also no access control per secret, no audit log of reads and no expiry. I use them as the last hop from a secret manager, or not at all with workload identity.

Q2Walk me through rotating a production database password without downtime.

I create a second credential, either a new password on a second database user or a second password if the database supports two, so both work at once. I publish it as the current version in the secret manager and roll the services so each picks it up. Once connection logs show nothing using the old credential, I revoke it. Automating exactly this cycle is what Secrets Manager rotation or Vault's database engine does.

Q3What happens when a developer pushes an AWS access key to a public GitHub repository?

Assume it is already being tried, because scanners harvest public pushes within minutes. I deactivate the key immediately, then check CloudTrail for what it did during the exposure, especially new IAM users, EC2 instances and S3 reads. Only then do I remove it from the code and add pre-commit and push scanning. Rewriting history alone does nothing, since forks and clones already have it.

Q4How do you let CI deploy to the cloud without storing credentials?

With OIDC federation. The CI platform issues a signed token describing the job, repository and branch; the cloud provider verifies it against a role's trust policy and returns credentials valid for about an hour. The trust policy must pin the repository and branch in the token subject, otherwise any repository on the platform could assume the role.

Q5What is envelope encryption and why use it?

Each piece of data is encrypted with its own data key, and that data key is encrypted with a master key that never leaves KMS. The app stores the encrypted data key alongside the data and only asks KMS to unwrap it when needed. It avoids sending large payloads to KMS, keeps the master key out of the app, and makes revocation, auditing and master-key rotation central.

Q6Are Kubernetes Secrets secure?

Only with extra configuration. By default they are base64 in etcd, so they rely entirely on who can reach etcd and its backups. I would enable encryption at rest with a KMS provider, restrict RBAC on Secrets tightly, and often sync from an external secret manager so rotation and auditing live there.

Q7How do you keep secrets out of a frontend bundle?

Nothing secret goes in a variable the bundler exposes, such as the NEXT_PUBLIC_ or VITE_ prefixes, because those are pasted into JavaScript that every visitor downloads. The browser calls my backend, and the backend holds the key. Publishable keys that are designed to be public, like Stripe's pk_ keys, are the only exception.

Key takeaways

Secrets leak from their copies: git history, image layers, logs, CI output and browser bundles. Deleting the source does not delete the copies.
Environment variables deliver secrets; a secret manager stores them with encryption, per-identity access, audit logs and rotation.
The best secret is none: IAM roles and OIDC federation give workloads credentials that expire within the hour.
Rotate with two valid credentials at once, switch consumers, verify the old one is unused, then revoke it.
Envelope encryption keeps the master key inside KMS and gives every record its own data key.
After a leak, rotate first, then audit, then clean up and add scanning. Rewriting history is not a fix.

Sources

#Context

#Why it matters

#Where secrets actually leak

#Environment variables vs secret managers

#Short-lived credentials instead of stored keys

#Rotation, envelope encryption and leak response