Not All Secrets Are Just Random Strings

The ticket was one line: rotate every secret in the app. There were more than 200 values in total. Most of them were third-party API keys with a clear rotation path: open the provider's dashboard, create a replacement, update the configuration, and move on.

For the values that looked like ordinary secrets, I used StackRoom's secret key generator, a developer tools utility in an app I am still building as a hobby project. I generated new values, pasted them into the .env, and moved on. I did not test the change in my local development environment. I was confident it would work, so I applied the changes to staging environment and started it.

That was the mistake. Staging broke, and tracing the failures led me to six values whose formats were not interchangeable. They looked like this:

ENCRYPTION_KEY=...
SSO_ENCRYPTION_KEY=...
SESSION_SECRET=...
JWT_SECRET=...
WEBHOOK_SECRET=...
INTERNAL_API_KEY=...

These 6 were scattered among 200+ secrets, same kind of name, no comments. Nothing in that file says that they follow three different sets of rules. It does not say that two can prevent the app from starting when their format is wrong, or that one also exists in a service I had not opened.

Untangling them took most of an afternoon.

Generating the value is the easy part#

I was right about one thing: a secret is random bytes. I was wrong to treat the encoding as decoration. It is part of the interface. Something downstream will turn that string back into bytes, using rules that are already fixed in the code.

After tracing the code where each value was used, the six fell into 3 groups.

Group 1: exactly 32 bytes, hex only#

ENCRYPTION_KEY and SSO_ENCRYPTION_KEY both feed AES-256-GCM through Node's crypto module. AES-256 requires a 256-bit key, or 32 bytes. The code reads them like this:

const key = Buffer.from(process.env.ENCRYPTION_KEY, "hex");

Because they are parsed as hex, each value must be exactly 64 characters and contain only valid hex digits. Two characters per byte, 32 bytes, 64 characters. Any other value is not a shorter key. It is a broken one.

That requirement was not obvious when I first saw the variables. I treated the encryption key and SSO encryption key like ordinary generated secrets. StackRoom did what I asked and produced random values. I had not checked which encoding this application expected.

This command always produces the right format:

openssl rand -hex 32

The one that failed quietly#

This is what took most of the afternoon, and why I am writing it down.

The value I generated was a perfectly good 32 bytes of randomness in a 44-character string. It was a strong secret with the right number of bytes and the wrong alphabet. You can reproduce the same kind of mismatch with openssl rand -base64 32.

Buffer.from(someBase64String, "hex") does not throw or warn. It decodes until it reaches a character that is not a hex digit, then returns whatever it decoded before that point.

I checked this on Node v22 rather than trusting my memory:

const b64 = crypto.randomBytes(32).toString("base64");
// 'YXWFbfAU08wV2fnVI813J/NxoUo2Fj7LytM9zsjldx8='  (44 characters)
 
Buffer.from(b64, "hex").length;
// 0

Zero bytes. The string starts with O, which is not a hex digit, so the decoder stops immediately and returns an empty buffer without complaint. A 44-character secret becomes nothing at all. The only sign is an empty Buffer several call frames away from the configuration value you were inspecting.

It is not always zero, which is worse. Decoding stops at the first bad character, so the amount that survives depends entirely on the string your random generator produced:

Buffer.from("abcdefgh", "hex").length; // 3   (stops at 'g')
Buffer.from("abc", "hex").length; // 1   (odd length, last digit dropped)

Then, eventually, you get this:

ERR_CRYPTO_INVALID_KEYLEN: Invalid key length

That error is misleading when you are looking at a 44-character string in your .env. The message says length, and the string is plainly long enough. The problem is the mismatch between the encoding used to write the value and the encoding used to read it. The length error is only the final symptom. I spent a while debugging it from the wrong end.

One encryption key was validated at startup and produced a clear error naming the variable and required length. The other was not, and that was the one that got me. They were the same class of secret, but only one had a guardrail. The .env gave no clue.

Group 2: the ones that don't care#

SESSION_SECRET, JWT_SECRET, and WEBHOOK_SECRET work differently. They sign things rather than encrypt them: session cookies, HS256 tokens, and HMAC-SHA256 webhook signatures. HMAC accepts a key of any length and internally pads or hashes it to fit the block size, so it genuinely does not care about the format you hand it.

Any long random string works. Base64, hex, alphanumeric, all fine.

That is what makes the first group easy to get wrong. Three of these six secrets accept almost anything, so it is natural to assume that all six are forgiving. The two encryption keys are not.

Group 3: the one that lives in two places#

INTERNAL_API_KEY has no format requirement. It is compared byte for byte with a constant-time comparison. It only needs to be long, unguessable, and identical on both sides.

The catch is the phrase "both sides." The main backend server has the key, and so do the Helper worker, in their own environment and a different Dokku application. Rotate it in one place but not the other and nothing fails at boot. Everything comes up green. The failure appears later, in an API call across those services, as an authentication error that looks unrelated to a configuration change.

Reading the backend .env could not reveal that requirement. The other half was not in the file or the repository.

The two environments don't even store secrets the same way. Locally, each runtime keeps its own .env file inside the codebase — one for the main backend server, a separate one for the Helper, since the two read configuration independently. Staging has no file at all: every value lives in Infisical, and the same set is injected into both of them at deploy time. That's precisely why INTERNAL_API_KEY has to match on both sides. In staging it's one value fanned out to two services, but locally it's two files that someone has to remember to keep in sync by hand.

What I'd actually change#

Fixing this once was not enough. The next full rotation will still involve more than 200 values, and these six will look just as uniform to the person handling it.

Validate at boot and fail loudly. Check every secret with a real constraint when the app starts. The error should name the variable, state the rule, and give the command that produces a valid value. ENCRYPTION_KEY must be 64 hex characters (openssl rand -hex 32), got 44 takes fifteen seconds to fix. Invalid key length from a crypto call at request time can take an afternoon.

When code reads a value with an explicit encoding, that encoding is part of the contract. Put that contract somewhere the next person will see before changing the value.

Put the contract where the team will find it#

I did not leave this as a comment above the variable. I wrote it up in Tribe Knowledge, where our team records what is not obvious from the code: the required format and length for each secret, the exact openssl command for every group, the fact that the internal key must be same in both services, and that everything needs a restart before picking up the new values.

What stays with me is how ordinary the failure was. Nobody wrote bad code. Buffer.from behaves exactly as documented, and the variable names are reasonable. Out of more than 200 values, six config lines looked identical but were not. The information that distinguished them was scattered across four source files and a repository I had not opened.

Rotating most secrets is easy. The work is spotting the few that have contracts hidden outside the configuration file.