4. Verify deployment

Owner: Cloud Ops · Time: ~30 min, plus certificate wait
You’ll finish with: every health check green, from inside the cluster and from a managed-device network.

Run the common checks, then the ingress and database checks for your cloud. Do not hand the endpoint to IT until it works from a network representative of managed devices.

Record the deployment identity

Before troubleshooting anything, write down:

• Cloud provider, account/project/subscription, and region.
• Terraform release tag and Helm chart version.
• Kubernetes cluster name and namespace.
• The HTTPS hostname.
• Whether GPU embedding, SCIM, and the fleet console are enabled.

This prevents commands from being run against the wrong environment.

Runtime and secret health

kubectl get pods,svc,ingress,cronworkflow -n arbium
kubectl get externalsecret -n arbium
kubectl get events -n arbium --sort-by='.lastTimestamp' | tail -30

Expected application pods:

• chaindb-edge-fns: ready replicas matching the configured baseline.
• embedder: one ready CPU or GPU pod.
• chaindb-scim: ready only when scim.enabled=true.
• admin-ui: ready only when admin.enabled=true.
• chaindb-capturelake: ready. This is where uploads are stored, so uploads fail while it is down.

Expected scheduled jobs (CronWorkflows) include chaindb-capturelake-derive, chaindb-capturelake-flush, chaindb-capturelake-compact, chaindb-capturelake-maintenance-full, and chaindb-capturelake-search-reconcile. Their runs appear as Workflows (kubectl get workflows -n arbium). If kubectl reports that it does not know the resource type cronworkflow, the workflow controller’s definitions are not installed: a first install applies them automatically, and an upgrade from an older release applies them by hand first, as in the “Upgrading from the CronJob engine” section of the chart’s INSTALL.md.

Every required ExternalSecret should report a successful sync. Confirm the runtime and pull Secrets exist without printing their values:

kubectl get secret chaindb-runtime ghcr-pull -n arbium
kubectl describe externalsecret -n arbium

If the Admin UI or SCIM is enabled, also expect admin-ui-oidc, admin-ui-session, and scim-bearer as configured.

Provider ingress and TLS

AWS

kubectl get ingress chaindb -n arbium
kubectl describe ingress chaindb -n arbium

The Ingress address should be an AWS ALB hostname. Your DNS record must point to that hostname, and the attached ACM certificate must be ISSUED in the same region as the ALB.

Google Cloud

kubectl get ingress chaindb -n arbium
kubectl get managedcertificate chaindb-tls -n arbium
kubectl describe managedcertificate chaindb-tls -n arbium

The Ingress address should match the Terraform static IP and DNS A record. Wait for the managed certificate to become Active; provisioning commonly takes 10–60 minutes after DNS resolves.

Azure

kubectl get ingress chaindb -n arbium
kubectl get certificate chaindb-tls -n arbium
kubectl describe certificate chaindb-tls -n arbium

The Ingress address should match the Terraform static IP and DNS A record. The certificate should report Ready=True and the chaindb-tls Secret should exist.

In-cluster health

kubectl run chaindb-health-check \
  -n arbium \
  --rm -i \
  --restart=Never \
  --image=curlimages/curl:8.11.1 \
  -- -fsS http://chaindb-edge-fns:8000/functions/v1/health

Expected response:

{ "ok": true }

A timestamp field may also be present.

External endpoint health

From a representative managed-device network:

curl -fsS https://arbium.<your-domain>/functions/v1/health
curl -fsS https://arbium.<your-domain>/functions/v1/auth-config

/health proves the application is reachable through the provider ingress and TLS. /auth-config proves clients can discover the configured Entra method.

Confirm the backend received the intended OIDC values:

kubectl -n arbium get configmap chaindb-config \
  -o jsonpath='{.data.OIDC_ISSUER_URL}{"\n"}{.data.OIDC_AUDIENCE}{"\n"}{.data.OIDC_ALLOWED_ALGS}{"\n"}'

Expected: your v2 Entra issuer URL, the bare Arbium Agent client GUID, and RS256.

SCIM endpoint

Skip this section when scim.enabled=false.

Use the deployed SCIM token from the approved secret channel:

BASE=https://arbium.<your-domain>/scim/v2
SCIM_TOKEN=<deployed-scim-token>

curl -i "$BASE/ServiceProviderConfig" \
  -H "authorization: Bearer $SCIM_TOKEN"

curl -i "$BASE/Schemas" \
  -H "authorization: Bearer $SCIM_TOKEN"

Expected: HTTP 200 with application/scim+json. An incorrect token should return 401, not redirect to Microsoft sign-in.

Fleet console

Skip this section when admin.enabled=false.

kubectl get pod -n arbium -l app.kubernetes.io/component=admin-ui
kubectl logs -n arbium deploy/admin-ui --tail=50
curl -fsSI https://arbium.<your-domain>/admin/api/healthz

The health endpoint should return 200. IT completes the signed-in browser test — it requires a provisioned user in your tenant.

Run the release check

The checks above show that the pods are up and the endpoints answer. The release check goes one step further: it uses the deployment the way a person’s device and an AI assistant do, and tells you whether that works. Run it after the first install, and before and after every upgrade (see Upgrading the chart).

A passing run means all three parts worked, from inside the cluster, through the same address your clients use:

Part	What it checks
Sign-in	The sign-in configuration that the Arbium Agent reads and the sign-in page are served, and requests that carry no credentials are refused. It does not sign anyone in.
Assistant connection	An AI assistant can connect with the check account’s connector token and sees the tool that reads uploaded data.
Upload and read-back	A few test uploads from the check account’s device are accepted, stored, and read back through the assistant connection.

It does not sign in through your identity provider, and it does not wait for the background work that builds summaries and search results from uploads.

The check can run in two ways. Pick the one that fits the install:

• As a Helm test (releaseSmoke.helmTest.enabled). Needs nothing beyond the chart. This is the simplest way and works on every install.
• As an Argo Workflow (releaseSmoke.enabled). Needs Argo Workflows, which the chart installs when argo-workflows.enabled is true.

Both run the same check and use the same check account.

The commands on this page use the namespace arbium and the Helm release name arbium, as the rest of this guide does. Replace them if you changed global.namespace or the release name.

Turn it on in your values

Add the check to the application values file, next to the settings from Configure & install Helm, and upgrade. For the Helm test:

rootsApi:
  enabled: true

capturelake:
  canary:
    rootsUrl: https://arbium.<your-domain>

releaseSmoke:
  helmTest:
    enabled: true

rootsUrl is the host that serves the assistant connection, normally the same HTTPS host as everything else. Until DNS and the certificate are in place, use the in-cluster addresses instead: rootsUrl: http://roots-api.arbium.svc.cluster.local:3000, and capturelake.canary.edgeUrl: http://chaindb-edge-fns.arbium.svc.cluster.local:8000 (the upload address defaults to your public address).

To run it as an Argo Workflow instead, turn on argo-workflows.enabled and releaseSmoke.enabled in place of releaseSmoke.helmTest.enabled. Do not enable Argo Workflows on a cluster that already runs its own Argo Workflows: the two would share cluster-wide definitions. On an existing install, Helm applies the Argo Workflows definitions only on helm install, so apply them from the chart you are about to install before the upgrade:

kubectl apply --server-side --force-conflicts -f chaindb-<release-version>/chaindb/crds/

argo-workflows:
  enabled: true

rootsApi:
  enabled: true

capturelake:
  canary:
    rootsUrl: https://arbium.<your-domain>

releaseSmoke:
  enabled: true

The Helm test and the Argo Workflow are independent switches, and both may be on. Do not turn the Helm test on for a pipeline that only renders the chart and applies the result (helm template | kubectl apply, some GitOps tools): it would run the check on every apply.

The first upgrade that turns the check on has no “before” run, because the check arrives with that upgrade. Run it after that upgrade, and from the next upgrade on, before and after.

Create the check account

The check needs an account of its own: a person’s device uploads test data, and an AI assistant reads it back. It must not be a real person’s account. A command that ships with the API deployment creates this account and its two tokens in one step. You run it once, from a machine with kubectl access to the cluster.

The command runs inside the API deployment’s pod, so the API must already be deployed by the upgrade in the previous section (rootsApi.enabled: true). Wait until it is rolled out:

kubectl rollout status -n arbium deploy/roots-api

If the command below answers Error: Cannot find module '/app/scripts/release-check-account.mjs', the chart you deployed predates it: upgrade the chart first, then run it again.

Then create the account and store its tokens in a Secret in one pipeline:

kubectl exec -i -n arbium deploy/roots-api -c roots-api -- \
  node scripts/release-check-account.mjs ensure \
  | kubectl apply --server-side -n arbium -f -

Use --server-side here: a plain kubectl apply copies the whole Secret, tokens included, into the Secret’s kubectl.kubernetes.io/last-applied-configuration annotation, which kubectl describe secret prints. A server-side apply does not create that annotation. It does rewrite one that is already there, though, so if the Secret was ever applied without --server-side, remove the copy once:

kubectl annotate secret capturelake-canary -n arbium kubectl.kubernetes.io/last-applied-configuration-

The command writes only the Secret to its output, so nothing is applied if it fails. Everything else it says (the account id, when the connector token expires, what it created and what it found already there) goes to your terminal as messages.

What ensure does:

• Creates one dedicated account, capturelake-canary@arbium.invalid by default (change it with --email). The account has no password, cannot sign in, and belongs to no organization. It sees only its own test data, never anyone else’s, and nobody else sees its data through an organization or team.
• Takes one license seat, with the same seat rule as enrolling a device. If the license has no free seat, the command says so and creates nothing; free a seat or ask Arbium for a larger license, then run it again.
• Sets the account’s search access to personal, so that the assistant connection works whatever search rules your organization uses.
• Marks the account so that its test data is not analysed. The check sends a little test data, and nobody needs it turned into summaries, so the marked account uses no AI tokens. If the account already carries a different value for this mark, for example one you set by hand, the command stops with a message naming it and changes nothing; remove that value (or set it to false), then run it again.
• Issues a device token and a connector token, and puts both in a Secret named capturelake-canary with the keys PROBE_DEVICE_TOKEN and PROBE_MCP_TOKEN. The connector token is valid for 90 days by default (--days <n> changes that).
• Refuses to continue if the account already exists and belongs to an organization. Use another --email.

The Secret name must match the chart’s capturelake.canary.tokensSecret, which is capturelake-canary unless you changed it. If you set another name there, pass the same name to the command with --secret-name <name>.

It is safe to run ensure again: it finds the account instead of creating a second one. Each run issues a new device token and a new connector token and replaces both in the Secret. Earlier device tokens keep working until you revoke them (below).

If you keep secrets in your cloud secret store and sync them into the cluster (externalSecrets.extraSecrets), do not apply the Secret directly under the name the sync owns. Run the command with a different --secret-name, for example capturelake-canary-bootstrap, copy the two values into your secret store under the names your sync expects, and delete the bootstrap Secret:

kubectl get secret capturelake-canary-bootstrap -n arbium \
  -o jsonpath='{.data.PROBE_DEVICE_TOKEN}' | base64 -d; echo
kubectl get secret capturelake-canary-bootstrap -n arbium \
  -o jsonpath='{.data.PROBE_MCP_TOKEN}' | base64 -d; echo
kubectl delete secret capturelake-canary-bootstrap -n arbium

Run it

As a Helm test. Helm waits 300 seconds by default, which is less than the check’s own limit (releaseSmoke.activeDeadlineSeconds, 600 by default), so always pass --timeout with that value plus 60 seconds. The install’s notes print the exact line:

helm test arbium -n arbium --logs --timeout 660s

A passing run ends with Phase: Succeeded and prints one line of results. It usually takes a few minutes, at most about ten. The test pod stays afterwards so you can read its log; the next run replaces it, and helm uninstall does not delete it (deleting the namespace does).

As an Argo Workflow. The chart ships the check as a Workflow template named chaindb-release-smoke. Create a run from it and wait for it to finish:

WF=$(kubectl create -n arbium -o name -f - <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: release-check-
spec:
  workflowTemplateRef:
    name: chaindb-release-smoke
YAML
)
echo "$WF"

for _ in $(seq 90); do
  PHASE=$(kubectl get -n arbium "$WF" -o jsonpath='{.status.phase}')
  case "$PHASE" in Succeeded|Failed|Error) break ;; esac
  sleep 10
done
echo "$PHASE"

Each run sends releaseSmoke.count (3 by default) real test uploads for the check account: safe to repeat, not to run in a loop.

Read the result

Helm test. helm test fails on any non-zero exit. The exit code of the test pod says why:

kubectl get pod chaindb-release-smoke-test -n arbium \
  -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'
kubectl logs chaindb-release-smoke-test -n arbium

Exit	Meaning	What to do
0	All three parts passed.	Nothing.
6, 7	The sign-in part (6) or the assistant connection (7) answered wrongly.	The release is broken. Run the check once more; after two failures in a row, treat it as a real failure.
3, 4, 5	A test upload was refused (3), did not arrive in time (4), or could not be read back (5).	Same as 6 and 7.
2	The check could not judge: wrong settings, or a proof it cannot make on this install.	Not a broken release. Fix the settings the message names, for example rootsUrl.
8	The connector token has expired or has less than a day left.	Not a broken release. Renew the connector token.
1	The check itself crashed.	Read the pod log.

A pod reason of DeadlineExceeded means the check hit its time limit. A pod stuck on CreateContainerConfigError means the Secret is missing or has the wrong name: helm test waits for the pod’s time limit before it fails, so check this first with kubectl get secret capturelake-canary -n arbium.

Argo Workflow. Succeeded means all three parts passed. Failed usually means a part did not pass. A run can also end Failed or Error with no result line, for example when the Secret is missing or the pod could not start (kubectl describe -n arbium "$WF" says why); that is a problem with the check, not a verdict on the release. For a run that did not succeed, read the result line the check printed:

kubectl logs -n arbium -l workflows.argoproj.io/workflow="${WF#*/}" -c main --tail=5

In both ways the result line lists the parts that failed in "failed", with a short reason in each part’s detail:

Name in failed	Part	Where to look
auth	Sign-in	The OIDC settings in chaindb-config and the ingress, as in External endpoint health.
mcp	Assistant connection	An expired or wrong connector token, and rootsUrl: the host must serve the assistant connection.
canary	Upload and read-back	The uploads service pod (chaindb-capturelake) and its logs, and an expired token: uploads fail while it is down.

A part marked "judged":false could not be evaluated, which points at the check’s own settings rather than the release. A single failure can be a passing blip: fix what the reason names or just run the check once more, and treat two failures in a row as a real failure. The line also carries token_days_left, the days left on the connector token. On the Argo path, the failed run’s pod stays for a week for this; delete a run you no longer need with kubectl delete -n arbium "$WF".

Renew the connector token

The connector token expires after 90 days by default, and an expired token makes the check fail. Renew it before it lapses: when token_days_left in the result line drops below 14, renew it. The check does not run at all when less than a day is left (exit 8 as a Helm test), so it never blames the release for an expired token.

kubectl exec -i -n arbium deploy/roots-api -c roots-api -- \
  node scripts/release-check-account.mjs rotate-connector \
  | kubectl patch secret capturelake-canary -n arbium --type merge --patch-file /dev/stdin

This issues a new connector token (--days <n> sets its lifetime) and replaces only PROBE_MCP_TOKEN in the Secret. The device token is unchanged and does not expire. If your Secret has another name, patch that one; if you sync it from your cloud secret store, put the new value in the store instead (see below). The command uses the default account email; pass --email if you created the account under another one.

For a Secret that External Secrets manages, do not patch it: run the command without the pipe, copy the PROBE_MCP_TOKEN value from the patch it prints into your secret store, and let the sync update the Secret:

kubectl exec -i -n arbium deploy/roots-api -c roots-api -- \
  node scripts/release-check-account.mjs rotate-connector

Look at the account, or switch it off

Show the account’s state. The output contains no tokens:

kubectl exec -i -n arbium deploy/roots-api -c roots-api -- \
  node scripts/release-check-account.mjs status

It reports the account id and email, how many organizations the account belongs to (it should be 0), how many device tokens are still valid, whether the personal search setting is in place, and whether the account is marked so that its test data is not analysed (captureAnalysisOff), and the mark exactly as stored (captureAnalysisMarker: null when there is none, otherwise the stored value, for example false or "false" in quotes). A stored value that is not plain false shows here even though captureAnalysisOff is false.

To cut off the account, for example when you retire the check or suspect a token leaked, revoke it:

kubectl exec -i -n arbium deploy/roots-api -c roots-api -- \
  node scripts/release-check-account.mjs revoke

This revokes every device token and every connector token of the account. It does not delete the account and does not free its license seat. The check fails until you run ensure again, which issues new tokens. Pass --email to each of these commands if you used another email.

Troubleshooting

External Secrets are not ready

kubectl describe clustersecretstore chaindb-secret-store
kubectl describe externalsecret -n arbium
kubectl logs -n external-secrets deploy/external-secrets --tail=200

Check the provider-specific identity:

• AWS: the eso service account IRSA annotation and IAM permission on the exact Secrets Manager ARNs.
• Google Cloud: the GSA annotation, Workload Identity binding, project ID, cluster name, and region.
• Azure: the managed-identity client ID, tenant ID, Key Vault URI, federated credential, and Key Vault Secrets User role.

Pods report ImagePullBackOff

kubectl get secret ghcr-pull -n arbium
kubectl describe pod -n arbium <pod-name>

Confirm the cloud registry secret contains valid JSON with username and token, the token has read:packages, and the provider overlay enables externalSecrets.imagePullSecret with target name ghcr-pull.

Application pods fail during migrations or startup

kubectl logs -n arbium deploy/chaindb-edge-fns -c migrations --tail=200
kubectl logs -n arbium deploy/chaindb-edge-fns -c edge-fns --tail=200

Check the database path for the selected provider:

Provider	Expected connection	Common failure
AWS	Direct Aurora endpoint with sslmode=require	Security group, stale/unauthorized RDS secret, or wrong endpoint
Google Cloud	arbium-cloud-sql-proxy Service with application SSL disabled	Proxy Workload Identity, connection name, or incorrectly encoded password
Azure	Private Flexible Server endpoint with sslmode=require	Private DNS/VNet connectivity, Key Vault sync, or TLS configuration

Migrations are idempotent. Do not disable migrations.runOnStartup to hide a database error; fix the underlying connection or migration failure.

Embedding pod is pending or restarting

kubectl describe pod -n arbium -l app.kubernetes.io/component=embedder
kubectl top nodes

• In CPU mode, increase general-node capacity or the node-pool maximum.
• In GPU mode, confirm a node advertises nvidia.com/gpu, matches the cloud preset’s selector, and has the expected taints and tolerations.
• Azure CPU mode is the baseline unless an AKS GPU pool was added explicitly.

Ingress has no address or HTTPS fails

Start with kubectl describe ingress chaindb -n arbium, then inspect the provider controller:

# AWS
kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=200

# Azure
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=200
kubectl logs -n cert-manager deployment/cert-manager --tail=200

For Google Cloud, inspect the Ingress events, static IP annotation, DNS A record, and ManagedCertificate domain status. For every provider, verify the certificate covers the exact configured hostname.

Fleet-console pod fails or sign-in redirects loop

kubectl logs -n arbium deploy/admin-ui --tail=200

• AADSTS50011: the Entra Web redirect URI must exactly match https://<host>/admin/api/auth/callback.
• consent_required: grant tenant admin consent for every delegated permission on the Arbium Admin application.
• invalid_client: rotate the Entra client secret, update the cloud secret store, wait for External Secrets to sync, and restart admin-ui.
• /forbidden: verify the user, admin group, and group membership were synced through SCIM and that the admin groups set with set_access_groups (or under Organization → Access) are Entra group object IDs.

Scheduled jobs fail

kubectl get workflows -n arbium --sort-by=.metadata.creationTimestamp | tail -10
kubectl logs -n arbium -l workflows.argoproj.io/workflow=<workflow-name> --all-containers --tail=100

If Workflows are missing or failing, check that the Argo workflow controller pod is running and read the failed Workflows’ pods (kubectl get workflows above lists them; the kubectl logs line shows why one failed). To pause the schedules while debugging, set capturelake.pipeline.suspend: true in your values file and run helm upgrade; set it back to false after health checks pass. A hand patch of a CronWorkflow can be reset by the next helm upgrade.

Next

Continue to 5. Handoff to IT.