--- description: Support ticket investigation. Parallel data gathering from DB, Datadog, OpsGenie, vtb, then hypothesis formation and user-guided exploration. category: verkada --- flowchart TD _HEADER_[""]:::headerStyle classDef headerStyle fill:none,stroke:none subgraph _MAIN_[" "] %% Phase 1: Setup subgraph Setup["Phase 1: Setup"] TICKET[Linear Support Ticket] --> CHECKOUT[Checkout Master
+ Ticket Branch] CHECKOUT --> MARK_IP[Mark In-Progress] end %% Phase 2: Context Gathering subgraph Context["Phase 2: Context Gathering (parallel)"] MARK_IP --> READ_TICKET[Read Ticket
+ All Comments] & READ_DOCS[Read Service Docs
+ Test Patterns] & RECENT_PRS[Search Recent PRs
Around Issue Timeline] READ_TICKET & READ_DOCS & RECENT_PRS --> CTX[Context Built] end %% Phase 3: Parallel Investigation subgraph Investigate["Phase 3: Parallel Investigation"] CTX --> DB_QUERY[DB Querying
Org + User State] & DATADOG[Datadog
Traces + Logs] & OPSGENIE[OpsGenie
Alerts + Incidents] & VTB[vtb
Org + Device Info] & VERKURL[verkurl
Replicate on Staging] DB_QUERY & DATADOG & OPSGENIE & VTB & VERKURL --> EVIDENCE[Collect Evidence] end %% Phase 4: Hypothesis subgraph Hypothesis["Phase 4: Hypothesis"] EVIDENCE --> THINK[Ultrathink:
Possible Causes] THINK --> ROUTES[Propose Investigation
Routes] ROUTES --> ASK{User Selects
Routes} end %% Phase 5: Explore subgraph Explore["Phase 5: Parallel Exploration"] ASK --> AGENTS[Spawn Sub-Agents
per Selected Route] AGENTS --> TDD[TDD Tests
Confirm Thesis] & DB_DEEP[Deep DB Analysis
Cross-Org Compare] & CODE_DIG[Code Archaeology
Blame + History] & STAGING[Staging Repro
via verkurl] TDD & DB_DEEP & CODE_DIG & STAGING --> RESULTS[Collect Findings] RESULTS --> REPORT[Report Results
+ Next Steps] end click TICKET "#" "**Linear Support Ticket**\nA support or bug ticket from Linear.\nContains: customer report, org ID, user ID, timestamps, error descriptions.\n\nPassed as `$ARGUMENTS` to `/v:investigate`." click CHECKOUT "#" "**Checkout Master + Ticket Branch**\nImmediately checkout and pull latest master,\nthen create/checkout the ticket branch based off master.\n\nThis branch is for tracking investigation work,\nnot necessarily for code fixes." click MARK_IP "#" "**Mark In-Progress**\nUpdate the Linear ticket status to in-progress.\nSignals to the team that investigation has started." click READ_TICKET "#" "**Read Ticket + Comments**\nRead the full Linear ticket:\n- Description with customer report\n- All comments (may have additional context)\n- Associated services mentioned\n- Org ID, user ID, timestamps\n\nRuns in parallel with other context reads." click READ_DOCS "#" "**Read Service Docs**\nRead documentation for the associated services:\n- `vcerberus/docs/` for access control\n- Service-specific `/docs` folders\n- Test files to understand intended behavior\n\nTests and docs imply correct intended behavior." click RECENT_PRS "#" "**Search Recent PRs**\nLook for code changes around the timeline the issue was reported.\n`gh pr list --search 'merged:>YYYY-MM-DD' --json ...`\n\nRecent deploys are a common root cause for new issues." click CTX "#" "**Context Built**\nAll parallel context reads complete.\nNow have: ticket details, service docs, and timeline of recent changes." click DB_QUERY "#" "**DB Querying**\nQuery the organization's shard database:\n- Org settings and configuration\n- User credentials and state\n- Door/schedule/access policy records\n- Compare against known-good orgs\n\nUse `verkada-db-querying` skill." click DATADOG "#" "**Datadog Traces + Logs**\nSearch Datadog for:\n- Error traces around the reported timestamps\n- Service logs for the affected org/user\n- Latency spikes or anomalies\n- Associated service dependencies\n\nUse `datadog-cli` skill." click OPSGENIE "#" "**OpsGenie Alerts**\nCheck for related:\n- Alerts fired around the issue timeline\n- Active incidents\n- On-call escalations\n- Recurring alert patterns\n\nUse `opsgenie-cli` skill." click VTB "#" "**vtb: Org + Device Info**\nGather information about:\n- The organization's configuration\n- Device firmware and settings\n- Feature flags enabled\n- Mobile NFC/Apple Wallet settings\n\nUse `vtb` (Verkada Toolbox) skill." click VERKURL "#" "**verkurl: Replicate on Staging**\nAttempt to replicate the issue:\n- Use existing verkurl profiles\n- Test against staging and production\n- Compare behavior between environments\n\nUse `verkurl` skill with authenticated requests." click EVIDENCE "#" "**Collect Evidence**\nGather findings from all 5 parallel investigation streams.\nCross-reference: do DB state, logs, and alerts tell a consistent story?" click THINK "#" "**Ultrathink: Possible Causes**\nUse extended thinking to reason through:\n- What could cause the observed symptoms?\n- Do recent code changes explain it?\n- Is it data/config-specific or systemic?\n- Are there multiple possible root causes?\n\nForm ranked hypotheses with supporting evidence." click ROUTES "#" "**Propose Investigation Routes**\nPresent concrete next steps to explore each hypothesis.\nExamples:\n- Write TDD tests to confirm a thesis\n- Deep-dive DB comparing affected vs healthy orgs\n- Git blame/history around suspicious code paths\n- Staging reproduction with specific parameters" click ASK "#" "**User Selects Routes**\nPresent routes via AskUserQuestion with checkboxes (not single-select).\nUser can pick multiple routes to explore in parallel.\n\nThis ensures investigation effort is directed, not open-ended." click AGENTS "#" "**Spawn Sub-Agents**\nLaunch one background agent per selected route.\nAll run in parallel for maximum throughput.\nEach agent focuses on one investigation angle." click TDD "#" "**TDD Tests**\nWrite targeted tests to confirm or disprove a hypothesis.\nIf the test fails as expected, the thesis is supported.\nIf it passes, the thesis is wrong -- valuable signal either way." click DB_DEEP "#" "**Deep DB Analysis**\nCompare the affected org's database state against healthy orgs.\nLook for abnormal settings, missing records, or corrupted state.\nSQL queries across the shard's PostgreSQL." click CODE_DIG "#" "**Code Archaeology**\nGit blame and history around suspicious code paths.\nTrace when behavior changed.\nCorrelate with deploy dates and the issue timeline." click STAGING "#" "**Staging Reproduction**\nTargeted reproduction via verkurl with specific parameters\nderived from the hypothesis.\nCompare staging vs production behavior." click RESULTS "#" "**Collect Findings**\nGather results from all parallel exploration agents.\nDo findings converge on a single root cause?\nOr do they reveal additional complexity?" click REPORT "#" "**Report Results + Next Steps**\nPresent to the user:\n- Confirmed/disproved hypotheses\n- Root cause (if identified)\n- Recommended fix or further investigation\n- Evidence supporting each conclusion\n\nUpdate the Linear ticket with findings." classDef setup fill:#d1ecf1,stroke:#7ec8d8 classDef context fill:#d1ecf1,stroke:#7ec8d8 classDef investigate fill:#dfe6ff,stroke:#5b7bce classDef hypothesis fill:#fff3cd,stroke:#f0c040 classDef explore fill:#e8daef,stroke:#b07cc6 classDef approved fill:#d4edda,stroke:#5cb85c class TICKET,CHECKOUT,MARK_IP setup class READ_TICKET,READ_DOCS,RECENT_PRS,CTX context class DB_QUERY,DATADOG,OPSGENIE,VTB,VERKURL,EVIDENCE investigate class THINK,ROUTES,ASK hypothesis class AGENTS,TDD,DB_DEEP,CODE_DIG,STAGING,RESULTS explore class REPORT approved end style _MAIN_ fill:none,stroke:none,padding:0 _HEADER_ ~~~ _MAIN_