4. Verify deployment

Owner: Cloud Ops

Run the common checks for every installation, then complete the ingress and database checks for the selected cloud. Do not hand the endpoint to IT until it works from a network representative of managed devices.

Record the deployment identity

Before troubleshooting, write down:

• Cloud provider, account/project/subscription, and region.
• Terraform release tag and Helm chart version.
• Kubernetes cluster name and namespace.
• Customer HTTPS hostname.
• Whether GPU embedding, SCIM, and the Admin UI are enabled.

This prevents commands from being run against the wrong customer environment.

Runtime and secret health

kubectl get pods,svc,ingress,cronjob -n arbium
kubectl get externalsecret -n arbium
kubectl get events -n arbium --sort-by='.lastTimestamp' | tail -30

Expected application pods:

• chaindb-edge-fns: ready replicas matching the configured baseline.
• embedder: one ready CPU or GPU pod.
• chaindb-scim: ready only when scim.enabled=true.
• admin-ui: ready only when admin.enabled=true.

Expected scheduler CronJobs include close-stale-sessions, drain-batch, and nightly-enqueue.

Every required ExternalSecret should report a successful sync. Confirm the runtime and pull Secrets exist without printing their values:

kubectl get secret chaindb-runtime ghcr-pull -n arbium
kubectl describe externalsecret -n arbium

If the Admin UI or SCIM is enabled, also expect admin-ui-oidc, admin-ui-session, and scim-bearer as configured.

Provider ingress and TLS

AWS

kubectl get ingress chaindb -n arbium
kubectl describe ingress chaindb -n arbium

The Ingress address should be an AWS ALB hostname. The customer DNS record must point to that hostname, and the attached ACM certificate must be ISSUED in the same region as the ALB.

Google Cloud

kubectl get ingress chaindb -n arbium
kubectl get managedcertificate chaindb-tls -n arbium
kubectl describe managedcertificate chaindb-tls -n arbium

The Ingress address should match the Terraform static IP and DNS A record. Wait for the managed certificate to become Active; provisioning commonly takes 10–60 minutes after DNS resolves.

Azure

kubectl get ingress chaindb -n arbium
kubectl get certificate chaindb-tls -n arbium
kubectl describe certificate chaindb-tls -n arbium

The Ingress address should match the Terraform static IP and DNS A record. The certificate should report Ready=True and the chaindb-tls Secret should exist.

In-cluster health

kubectl run chaindb-health-check \
  -n arbium \
  --rm -i \
  --restart=Never \
  --image=curlimages/curl:8.11.1 \
  -- -fsS http://chaindb-edge-fns:8000/functions/v1/health

Expected response:

{ "ok": true }

A timestamp field may also be present.

External endpoint health

From a representative managed-device network:

curl -fsS https://arbium.<customer-domain>/functions/v1/health
curl -fsS https://arbium.<customer-domain>/functions/v1/auth-config

/health proves the application is reachable through the provider ingress and TLS. /auth-config proves clients can discover the configured Entra method.

Confirm the backend received the intended OIDC values:

kubectl -n arbium get configmap chaindb-config \
  -o jsonpath='{.data.OIDC_ISSUER_URL}{"\n"}{.data.OIDC_AUDIENCE}{"\n"}{.data.OIDC_ALLOWED_ALGS}{"\n"}'

Expected: the customer’s v2 Entra issuer URL, the bare Arbium Agent client GUID, and RS256.

SCIM endpoint

Skip this section when scim.enabled=false.

Use the deployed SCIM token from the approved secret channel:

BASE=https://arbium.<customer-domain>/scim/v2
SCIM_TOKEN=<deployed-scim-token>

curl -i "$BASE/ServiceProviderConfig" \
  -H "authorization: Bearer $SCIM_TOKEN"

curl -i "$BASE/Schemas" \
  -H "authorization: Bearer $SCIM_TOKEN"

Expected: HTTP 200 with application/scim+json. An incorrect token should return 401, not redirect to Microsoft sign-in.

Admin UI

Skip this section when admin.enabled=false.

kubectl get pod -n arbium -l app.kubernetes.io/component=admin-ui
kubectl logs -n arbium deploy/admin-ui --tail=50
curl -fsSI https://arbium.<customer-domain>/admin/api/healthz

The health endpoint should return 200. IT completes the signed-in browser test because it requires a provisioned user in the customer tenant.

Troubleshooting

External Secrets are not ready

kubectl describe clustersecretstore chaindb-secret-store
kubectl describe externalsecret -n arbium
kubectl logs -n external-secrets deploy/external-secrets --tail=200

Check the provider-specific identity:

• AWS: the eso service account IRSA annotation and IAM permission on the exact Secrets Manager ARNs.
• Google Cloud: the GSA annotation, Workload Identity binding, project ID, cluster name, and region.
• Azure: the managed-identity client ID, tenant ID, Key Vault URI, federated credential, and Key Vault Secrets User role.

Pods report ImagePullBackOff

kubectl get secret ghcr-pull -n arbium
kubectl describe pod -n arbium <pod-name>

Confirm the cloud registry secret contains valid JSON with username and token, the token has read:packages, and the provider overlay enables externalSecrets.imagePullSecret with target name ghcr-pull.

Edge functions fail during migrations or startup

kubectl logs -n arbium deploy/chaindb-edge-fns -c migrations --tail=200
kubectl logs -n arbium deploy/chaindb-edge-fns -c edge-fns --tail=200

Check the database path for the selected provider:

Provider	Expected connection	Common failure
AWS	Direct Aurora endpoint with sslmode=require	Security group, stale/unauthorized RDS secret, or wrong endpoint
Google Cloud	arbium-cloud-sql-proxy Service with application SSL disabled	Proxy Workload Identity, connection name, or incorrectly encoded password
Azure	Private Flexible Server endpoint with sslmode=require	Private DNS/VNet connectivity, Key Vault sync, or TLS configuration

Migrations are idempotent. Do not disable migrations.runOnStartup to hide a database error; fix the underlying connection or migration failure.

Embedder is pending or restarting

kubectl describe pod -n arbium -l app.kubernetes.io/component=embedder
kubectl top nodes

• In CPU mode, increase general-node capacity or the node-pool maximum.
• In GPU mode, confirm a node advertises nvidia.com/gpu, matches the cloud preset’s selector, and has the expected taints and tolerations.
• Azure CPU mode is the baseline unless an AKS GPU pool was added explicitly.

Ingress has no address or HTTPS fails

Start with kubectl describe ingress chaindb -n arbium, then inspect the provider controller:

# AWS
kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=200

# Azure
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=200
kubectl logs -n cert-manager deployment/cert-manager --tail=200

For Google Cloud, inspect the Ingress events, static IP annotation, DNS A record, and ManagedCertificate domain status. For every provider, verify the certificate covers the exact configured hostname.

Admin UI pod fails or sign-in redirects loop

kubectl logs -n arbium deploy/admin-ui --tail=200

• AADSTS50011: the Entra Web redirect URI must exactly match https://<host>/admin/api/auth/callback.
• consent_required: grant tenant admin consent for every delegated permission on the Arbium Admin application.
• invalid_client: rotate the Entra client secret, update the cloud secret store, wait for External Secrets to sync, and restart admin-ui.
• /forbidden: verify the user, admin group, and group membership were synced through SCIM and that scim.scimAdminGroupIds contains Entra group object IDs.

Scheduler CronJobs fail

kubectl logs -n arbium -l app.kubernetes.io/component=scheduler --tail=100

Confirm the scheduler token synced into chaindb-runtime and that the in-cluster edge-functions Service is reachable. Suspend the CronJobs while debugging and resume them after health checks pass.

Next

Continue to 5. Handoff to IT.