--- description: Verkada backend service build and deploy workflow using vdev/just. Covers repo selection, build modes (CREW/invoke/local), deployment to staging/prod, and monitoring via ArgoCD and vdog. category: verkada --- flowchart TD _HEADER_[""]:::headerStyle classDef headerStyle fill:none,stroke:none subgraph _MAIN_[" "] %% Phase 1: Preparation subgraph Prep["Phase 1: Preparation"] FIND_REPO[Find Free Repo
via repos] --> PULL[Pull Latest Master] PULL --> AUTOFIX[Run just autofix] AUTOFIX --> AWS_CHECK{AWS Creds
Valid?} AWS_CHECK -->|No| LOGIN[just login] AWS_CHECK -->|Yes| READY[Ready to Build] LOGIN --> READY end %% Phase 2: Build subgraph Build["Phase 2: Build"] READY --> MODE{Build Mode?} MODE -->|Default| CREW[CREW Remote Build
just build -c SERVICE] MODE -->|Invoke| INVOKE[Invoke Build
--use-invoke] MODE -->|Local| LOCAL[Local Bazel Build
--local] CREW --> BUILD_RESULT{Build OK?} INVOKE --> BUILD_RESULT LOCAL --> BUILD_RESULT BUILD_RESULT -->|Yes| HASH[Extract Build Hash] BUILD_RESULT -->|No| BUILD_DIAG{Diagnose Failure} BUILD_DIAG -->|Retry invoke| INVOKE BUILD_DIAG -->|Retry local| LOCAL BUILD_DIAG -->|Auth error| LOGIN end %% Phase 3: Deploy subgraph Deploy["Phase 3: Deploy"] HASH --> ENV{Target Environment?} ENV -->|Staging| DEPLOY_STAGE[Deploy to Staging
-e staging] ENV -->|Prod| APPROVE{Team Approval?} APPROVE -->|Yes| DEPLOY_PROD[Deploy to Prod
-e prod] APPROVE -->|No| WAIT_APPROVE[Get Approval] WAIT_APPROVE --> APPROVE DEPLOY_STAGE --> DEPLOY_RESULT{Deploy OK?} DEPLOY_PROD --> DEPLOY_RESULT DEPLOY_RESULT -->|Yes| MONITOR_START[Start Monitoring] DEPLOY_RESULT -->|No| DEPLOY_DIAG[Check ArgoCD
for Errors] DEPLOY_DIAG --> DEPLOY_FIX{Fix Type?} DEPLOY_FIX -->|Sync issue| ARGO_SYNC[ArgoCD Sync/Prune] DEPLOY_FIX -->|Code issue| READY ARGO_SYNC --> MONITOR_START end %% Phase 4: Monitoring subgraph Monitor["Phase 4: Monitor"] MONITOR_START --> ARGO_STATUS[ArgoCD App Status] MONITOR_START --> VDOG[vdog Pod Versions
by version tag] ARGO_STATUS --> ROLLOUT{Rollout
Complete?} VDOG --> ROLLOUT ROLLOUT -->|Yes| DONE([Deploy Complete]) ROLLOUT -->|No: stuck| ARGO_WAIT[Wait + Re-check] ROLLOUT -->|No: failing| ROLLBACK[Rollback via ArgoCD] ARGO_WAIT --> ROLLOUT ROLLBACK --> READY end %% CI/CD Alternate Path subgraph CICD["Alternate: CI/CD Pipeline"] CI_START[CI Trigger] --> CI_BUILD[just build -c SERVICE
-m text] CI_BUILD --> CI_HASH[Parse BUILD NUMBER
from stdout] CI_HASH --> CI_DEPLOY[just deploy
-m text] CI_DEPLOY --> CI_DONE([Pipeline Complete]) end click FIND_REPO "#" "**Find Free Repo**\nRun `repos` to see status of all Verkada-Backend-{1-10} directories.\nLook for: `FREE (master, clean)` in green.\n\nA free repo means:\n- On master with no branch checked out\n- Clean working directory\n- No unpushed commits" click PULL "#" "**Pull Latest Master**\nNavigate to the free repo and update:\n`cd ~/verkada/Verkada-Backend-N`\n`git pull origin master`\n\nCritical for clean builds -- stale code causes:\n- Build failures from outdated deps\n- Inconsistent artifacts\n- Deployment issues" click AUTOFIX "#" "**Run just autofix**\nFix linting, formatting, and BUILD files before building.\nMandatory step to avoid CI failures.\n\n`just autofix`" click AWS_CHECK "#" "**AWS Credentials Valid?**\nBuilds require valid AWS credentials for:\n- S3 artifact storage\n- ECR container registry\n- BuildBuddy remote execution\n\nIf expired, run `just login` to re-authenticate via SSO." click LOGIN "#" "**just login**\nPerform AWS SSO login.\nRequired when credentials expire (typically every 8-12 hours).\n\n`just login`" click READY "#" "**Ready to Build**\nAll prerequisites met:\n- Free repo on latest master\n- Autofix applied\n- AWS credentials valid\n\nProceed to choose build mode." click MODE "#" "**Build Mode Selection**\n- **Default (auto):** Uses CREW remote build if available, else falls back to local. Best for most development.\n- **Invoke:** Force local build via invoke. Use with `--publish` for artifact publishing.\n- **Local:** Force local bazel build. Best for quick iterations.\n\nThere is NO `--build-mode` flag. Use `--use-invoke` or `--local`." click CREW "#" "**CREW Remote Build**\nDefault build mode using remote CREW infrastructure.\n`just build -c SERVICE_NAME`\n\nFull-screen TUI with progress bars and spinners.\nAdd `-m text` for plain text output.\n\nReturns a build hash on success." click INVOKE "#" "**Invoke Build**\nLocal build via invoke, supports publishing.\n`just build -c SERVICE_NAME --use-invoke`\n\nUse with `--publish` to push artifacts.\nRequired for local builds that need to publish." click LOCAL "#" "**Local Bazel Build**\nDirect local bazel build, fastest for iteration.\n`just build -c SERVICE_NAME --local`\n\nNo remote infrastructure needed.\nBest for quick local testing cycles." click BUILD_RESULT "#" "**Build Result**\nSuccess: Build hash printed to output.\nFailure: Check error messages for diagnosis.\n\nIn text mode, look for:\n`BUILD NUMBER: abc123def456`" click HASH "#" "**Extract Build Hash**\nCapture the build hash for deployment.\n\nInteractive: note the hash from TUI output.\nCI/CD: parse from text output:\n`grep 'BUILD NUMBER' build.log | awk '{print $NF}'`\n\nHash format: 12-char hex string (e.g., `abc123def456`)." click BUILD_DIAG "#" "**Diagnose Build Failure**\nCommon failures:\n- **Component not found:** verify name and spelling\n- **Remote build failed:** retry with `--use-invoke` or `--local`\n- **Auth error:** run `just login`\n- **BuildBuddy errors:** check logs via buildbuddy-cli" click ENV "#" "**Target Environment**\nChoose deployment target:\n- **staging:** Test environment, safe for experimentation\n- **prod:** Production, requires team approval\n\nCan also target specific shards with `-s` flag:\n`prod1`, `prod2`, `staging2`, etc." click DEPLOY_STAGE "#" "**Deploy to Staging**\n`just deploy --component SERVICE -e staging -b BUILD_HASH`\n\nSafe for testing. No approval required.\nAdd `-m text` for CI/CD pipelines.\n\nMonitor with ArgoCD after deploy." click APPROVE "#" "**Team Approval Required**\nProduction deployments require team approval.\nVerify:\n- CI is green\n- Staging tests passed\n- No conflicting deployments in progress\n- Team lead or on-call has approved" click DEPLOY_PROD "#" "**Deploy to Production**\n`just deploy --component SERVICE -e prod -b BUILD_HASH`\n\nFor specific shards:\n`just deploy --component SERVICE -s prod1 -b BUILD_HASH`\n\nAdditional flags:\n- `--dry-run` -- preview without creating PRs\n- `--no-auto-merge` -- require manual merge" click WAIT_APPROVE "#" "**Get Approval**\nRequest approval from team lead or on-call.\nInclude:\n- PR link with CI status\n- Staging test results\n- Summary of changes\n- Risk assessment" click DEPLOY_RESULT "#" "**Deploy Result**\nSuccess: deployment PR created and merged (or auto-merged).\nFailure: check ArgoCD for sync errors.\n\nCommon issues:\n- ArgoCD sync failures (often need pruning)\n- Conflicting deployments\n- Resource limits exceeded" click DEPLOY_DIAG "#" "**Check ArgoCD for Errors**\n`just argo app get SERVICE-ENVIRONMENT`\n\nLook for:\n- Sync status and health\n- Failed resources\n- Out-of-sync manifests\n- Pruning needed" click DEPLOY_FIX "#" "**Fix Type**\nDetermine if the deploy failure is:\n- **Sync issue:** ArgoCD needs sync/prune\n- **Code issue:** actual code problem, needs rebuild" click ARGO_SYNC "#" "**ArgoCD Sync/Prune**\nForce sync with pruning:\n`just argo app sync SERVICE-ENVIRONMENT --prune`\n\nMost stuck auto-deploys are caused by lack of pruning.\nCheck vsuiteweb/ArgoCD/auto-deploy if issues persist." click MONITOR_START "#" "**Start Monitoring**\nAfter successful deploy, monitor the rollout.\nUse both ArgoCD and vdog in parallel to track progress." click ARGO_STATUS "#" "**ArgoCD App Status**\nCheck deployment health:\n`just argo app get SERVICE-ENVIRONMENT`\n\nWatch progress:\n`just argo app wait SERVICE-ENVIRONMENT`\n\nView history:\n`just argo app history SERVICE-ENVIRONMENT`" click VDOG "#" "**vdog Pod Versions**\nTrack running pods by version during rollout:\n`vdog metrics query 'sum:kubernetes.pods.running{env:ENV,service:SERVICE*} by {version}' --from 5m`\n\nThe version tag shows the git commit SHA.\nUse to verify new version rolling out and old version scaling down." click ROLLOUT "#" "**Rollout Complete?**\nComplete: all pods running new version, old version scaled to zero.\nStuck: mixed versions not converging -- wait and re-check.\nFailing: pods crashing or health checks failing -- rollback." click DONE "#" "**Deploy Complete**\nService is live on the target environment.\nAll pods running the expected version.\n\nNext steps:\n- Monitor Datadog for error spikes\n- Check service health endpoints\n- Verify key functionality" click ARGO_WAIT "#" "**Wait and Re-check**\nRollout in progress but not yet complete.\nRe-check in 1-2 minutes.\n\nUse `just argo app wait SERVICE-ENVIRONMENT` for blocking wait." click ROLLBACK "#" "**Rollback via ArgoCD**\nPods are failing after deploy.\nRollback to previous known-good version.\n\nInvestigate root cause, fix, and re-enter the build phase." click CI_START "#" "**CI Trigger**\nAlternate entry for automated CI/CD pipelines.\nAll commands use `-m text` for parseable output.\nNo TUI rendering, no interactive prompts." click CI_BUILD "#" "**CI Build (Text Mode)**\nBuild in text mode for parseable output:\n`just build -c SERVICE -m text`\n\nCapture output to log file:\n`just build -c SERVICE -m text > build.log`\n\nDefault mode uses remote CREW when available." click CI_HASH "#" "**Parse Build Hash**\nExtract build hash from text output:\n`grep 'BUILD NUMBER' build.log | awk '{print $NF}'`\n\nValidate hash is non-empty before deploying.\nStore in environment variable for next step." click CI_DEPLOY "#" "**CI Deploy (Text Mode)**\nDeploy using extracted build hash:\n`just deploy --component SERVICE -e staging -b BUILD_HASH -m text`\n\nAlways validate build hash and check exit codes." click CI_DONE "#" "**Pipeline Complete**\nCI/CD pipeline finished.\nBuild artifacts published, deployment triggered.\nMonitor via ArgoCD and vdog." classDef prep fill:#d1ecf1,stroke:#7ec8d8 classDef build fill:#e8daef,stroke:#b07cc6 classDef deploy fill:#dfe6ff,stroke:#5b7bce classDef monitor fill:#fff3cd,stroke:#f0c040 classDef cicd fill:#ffeaa7,stroke:#e0c040 classDef success fill:#d4edda,stroke:#5cb85c classDef error fill:#f8d7da,stroke:#e06070 classDef decision fill:#fff3cd,stroke:#f0c040 class FIND_REPO,PULL,AUTOFIX,AWS_CHECK,LOGIN,READY prep class MODE,CREW,INVOKE,LOCAL,BUILD_RESULT,HASH,BUILD_DIAG build class ENV,DEPLOY_STAGE,APPROVE,DEPLOY_PROD,WAIT_APPROVE,DEPLOY_RESULT,DEPLOY_DIAG,DEPLOY_FIX,ARGO_SYNC deploy class MONITOR_START,ARGO_STATUS,VDOG,ROLLOUT,ARGO_WAIT monitor class CI_START,CI_BUILD,CI_HASH,CI_DEPLOY cicd class DONE,CI_DONE success class ROLLBACK error end style _MAIN_ fill:none,stroke:none,padding:0 _HEADER_ ~~~ _MAIN_