Runbooks / Secret Rotation

Secret Rotation

Operator runbook for rotating every long-lived credential or signed artefact in an IdentityMesh deployment. Pair with secrets-and-dpapi.md for the storage model.

The full inventory:

SecretWhere it livesRotation cadence (suggested)
License key%CommonApplicationData%\IdentityMesh\license.keyOn expiry / tier change
Relay agent API keyRelay:ApiKey in appsettings.json on each relay host90 days
SQL connection-string credentialsIdentityMesh:SqlConnectionString (Admin API + Sync Engine)90 days, or per AD policy
Connector credentials (LDAP, SQL, SaaS auth tokens)IM_Secrets table (DPAPI-encrypted)Per source-system policy
Service account passwords (SVC_ACCOUNT, API_SVC_ACCOUNT)Windows account / installed service identityPer AD policy
Code-signing certificateOperator’s certificate store, configured by thumbprintPer CA expiry
DPAPI master key (implicit)Local Windows profile / machineDon’t rotate — re-provision

Each rotation procedure below is idempotent and can be re-run if interrupted.


License key

Rotate when the key is reissued by sales (tier change, expiry extension, customer rename) or when you suspect the file leaked.

# 1. Place the new license file alongside the old one.
$dataDir = "$env:ProgramData\IdentityMesh"
Copy-Item -Path "C:\Path\To\new-license.key" `
          -Destination "$dataDir\license.key.new"

# 2. Atomic swap — replace the live file in one move so a
#    concurrent license check never sees a torn read.
Move-Item -Path "$dataDir\license.key.new" `
          -Destination "$dataDir\license.key" `
          -Force

# 3. Hot-reload via the Admin API. No service restart required —
#    the FileLicenseProvider re-reads on demand.
Invoke-RestMethod -Method Post `
                  -Uri https://<admin-host>/api/license/reload `
                  -UseDefaultCredentials

If you don’t have admin-API access, restart the IdentityMesh services to force the reload:

sc stop  IdentityMeshAdmin
sc stop  IdentityMeshEngine
sc start IdentityMeshAdmin
sc start IdentityMeshEngine

Verify by hitting GET /api/licenseTier, CustomerName, and ExpiresUtc should reflect the new file.

The license file is not a secret in the cryptographic sense — it’s an RSA-signed payload — but treat it as customer data. Never email it; never commit it.


Relay agent API key

Rotates the shared secret the relay uses to authenticate to the Admin API’s SignalR hub. The relay refuses to start with an empty Relay:ApiKey outside Development.

Phase 1 — issue the new key

There is no Admin API endpoint to mint relay keys today; pick a strong random value (≥ 32 bytes, base64) out-of-band:

# Cryptographically random 32-byte key, base64-encoded.
[Convert]::ToBase64String((1..32 | %{ [byte](Get-Random -Min 0 -Max 256) }))

Phase 2 — roll the relay (per host)

$cfg = "$env:ProgramFiles\IdentityMeshRelayAgent\appsettings.json"

# 1. Stop the relay so the file write is uncontended.
sc stop IdentityMeshRelayAgent

# 2. Update the ApiKey field. Use a JSON-aware editor; the
#    snippet below works for the simple flat schema.
$json = Get-Content $cfg -Raw | ConvertFrom-Json
$json.Relay.ApiKey = '<new-key>'
$json | ConvertTo-Json -Depth 10 | Set-Content $cfg -Encoding UTF8

# 3. Start the relay. It will reconnect to the hub with the new
#    key on first attempt; check the relay log for "Connected".
sc start IdentityMeshRelayAgent

Phase 3 — confirm + invalidate the old key

The Admin API maintains the accepted-key list (today: a single shared value in its own configuration). Update it to match, then remove the old value:

$cfg = "$env:ProgramFiles\IdentityMesh\appsettings.json"
$json = Get-Content $cfg -Raw | ConvertFrom-Json
$json.Relay.ApiKey = '<new-key>'
$json | ConvertTo-Json -Depth 10 | Set-Content $cfg -Encoding UTF8

sc stop  IdentityMeshAdmin
sc start IdentityMeshAdmin

Verify via GET /api/relays (requires dashboard.read) — the relay should show Connected = true with a recent heartbeat.

Note. Today’s relay key is a single value, so rotation requires a brief desync window where the relay is down. A short-lived-token replacement is on the roadmap.


SQL connection-string credentials

Two cases — pick the matching procedure.

Rotate the SQL service account’s domain password per your AD policy. No IdentityMesh-side action is needed; EF Core picks up the new ticket on its next connection.

If you change the service account (not just the password), re-run the installer’s Repair flow or update the service login manually:

sc config IdentityMeshAdmin obj= "DOMAIN\new-svc-account" password= "<password>"
sc config IdentityMeshEngine obj= "DOMAIN\new-svc-account" password= "<password>"
sc stop IdentityMeshAdmin; sc start IdentityMeshAdmin
sc stop IdentityMeshEngine; sc start IdentityMeshEngine

Case B — SQL auth (User ID=…;Password=…)

# 1. Update the connection string in BOTH appsettings.json files.
foreach ($cfg in @(
    "$env:ProgramFiles\IdentityMesh\appsettings.json",
    "$env:ProgramFiles\IdentityMesh.Service\appsettings.json"
)) {
    $json = Get-Content $cfg -Raw | ConvertFrom-Json
    $json.IdentityMesh.SqlConnectionString = '<new-cs>'
    $json | ConvertTo-Json -Depth 10 | Set-Content $cfg -Encoding UTF8
}

# 2. Cycle services together so a half-rotated state never runs.
sc stop IdentityMeshEngine
sc stop IdentityMeshAdmin
sc start IdentityMeshAdmin
sc start IdentityMeshEngine

Verify with /health/ready (returns 200 only when the DB check passes) and GET /api/dashboard.


Connector credentials in IM_Secrets

These are the per-connector passwords / API tokens that SqlSecretStore holds DPAPI-encrypted.

# Push a new value through the SecretsCli tool. Run on the
# Sync Engine host so it's encrypted under the right local
# machine key.
secretscli set "secret://ad/svc-imadmin/password"  "<new-password>"
secretscli set "secret://sql/hr-source/password"   "<new-password>"

# Verify
secretscli list

The change is picked up on the next sync run for the affected connector; the running engine reads through ISecretStore per connector activation, so no restart needed.

To find every secret reference an operator might need to rotate:

SELECT s.SecretRef, c.Name AS ConnectorName, s.UpdatedOn
FROM IM_Secrets s
LEFT JOIN IM_Connectors c
       ON CAST(c.ConnectorId AS NVARCHAR(36)) = SUBSTRING(s.SecretRef, …)
ORDER BY s.UpdatedOn ASC;

Host migration is NOT a rotation. DPAPI blobs only decrypt on the host that wrote them. If you’re moving to a new host, see the DR section of backup-and-restore.md — every secret must be re-pushed via secretscli set from the new host. The blobs in SQL travel; the keys do not.


Code-signing certificate

Used by the installer to sign the MSI, the service binaries, and the connector DLLs that the relay verifies before loading.

When the cert is approaching CA expiry (typically 1–3 years):

# 1. Import the new cert into the operator's signing store.
Import-PfxCertificate -FilePath "C:\path\to\new.pfx" `
                      -CertStoreLocation Cert:\CurrentUser\My `
                      -Password (ConvertTo-SecureString '<pwd>' -AsPlainText -Force)

# 2. Update the build script's CODESIGN_THUMBPRINT (or the
#    matching env var on the build host) to the new thumbprint.
#    See identitymesh-installer/build-installer.cmd.

# 3. Rebuild the installer + connector DLLs against the new cert.
.\build-installer.cmd

# 4. Update the relay's accepted-thumbprint config so already-
#    deployed relays accept connector DLLs signed with the new
#    cert. Updates land in IdentityMeshRelayAgent/appsettings.json
#    on each relay host.

The relay rejects DLLs signed with a thumbprint not on its accepted list, so plan a window during which both old and new thumbprints are accepted to allow rolling rebuild + redeploy.


What you can’t rotate


Cadence summary

FrequencyItems
Per AD policy (e.g. 90 days)Service-account passwords, SQL auth (if used)
90 daysRelay API key
Per source-system policyConnector credentials in IM_Secrets
Annual / per CA expiryCode-signing cert
As neededLicense key (issued by sales)

Document the dates of the last rotation in your CMDB or change-log of choice — IdentityMesh tracks IM_Secrets.UpdatedOn so you can see when each connector secret was last touched, but everything else is operator-owned.