SPOTTED

What SPOTTED does

SPOTTED watches a camera or video source, tracks the people and objects in frame, and surfaces possible concealment events for a human to review. When a reviewer confirms one, the system generates a spoken de-escalation announcement locally and plays it in the store. Every event flows to a database and an analytics warehouse, so a manager can ask questions about the event history in plain language.

Highlights

• Award winner. Best Use of Gemma 4, a Google-sponsored award, at HackABull 2026.
• No false accusations by design. A staff member reviews every flagged clip before anything happens.
• Private by default. Video is processed on site and never uploaded anywhere.
• Answers in plain English. Managers can ask questions about past events and get real answers.

The stack

How the pieces fit together

A Python backend does all frame processing on the host and emits lightweight event records. The Next.js dashboard, the announcement loop, the cloud clip review, and the analytics warehouse all work from those records and exported clips. Raw video never leaves the machine.

%%{init: {'theme':'base','themeVariables':{'background':'#0c0c0e','primaryColor':'#151517','primaryTextColor':'#e6e6e6','primaryBorderColor':'#00ff41','lineColor':'#555','secondaryColor':'#111','tertiaryColor':'#0a0a0a','clusterBkg':'#0c0c0e','clusterBorder':'#262626','fontFamily':'JetBrains Mono, monospace','fontSize':'13px','textColor':'#c0c0c0'}}}%%
flowchart TB
  subgraph EDGE["Local host (heavy pixels stay here)"]
    CAM["Camera / video"]
    CV["YOLOv8 + ByteTrack
ResNet re-ID + concealment heuristic"]
  end
  subgraph BACK["Backends"]
    ST["FastAPI stream
MJPEG + event writes"]
  end
  subgraph STORE["Event store"]
    MG[("MongoDB
events · alerts · cameras")]
  end
  subgraph WEB["Next.js 16 dashboard (R3F)"]
    OPS["Operator review · alerts · events"]
    ANA["Analytics chat"]
  end
  subgraph LOOP["Announce loop"]
    GEM["Gemma 4 (Ollama)
draft announcement"]
    EL["ElevenLabs TTS
speak deterrent"]
  end
  subgraph CLOUD["Cloud (candidate clips + analytics)"]
    GEMINI["Gemini 2.5
clip verdict (opt-in)"]
    SNOW[("Snowflake Cortex
Search + Complete")]
  end
  CAM --> CV --> ST --> MG
  MG --> OPS
  OPS -->|"confirm"| GEM --> EL
  OPS -->|"export clip"| GEMINI
  MG -->|"sync"| SNOW --> ANA

Fig. 1 · Component architecture. Only event records and exported clips ever leave the host.

The detection-to-deterrent loop

The live path is a chain of models feeding a heuristic, gated by a human before anything is spoken.

%%{init: {'theme':'base','themeVariables':{'background':'#0c0c0e','primaryColor':'#151517','primaryTextColor':'#e6e6e6','primaryBorderColor':'#00ff41','lineColor':'#555','actorBkg':'#151517','actorBorder':'#00ff41','actorTextColor':'#e6e6e6','signalColor':'#666','signalTextColor':'#c0c0c0','noteBkgColor':'#111','noteTextColor':'#c0c0c0','noteBorderColor':'#333','fontFamily':'JetBrains Mono, monospace','fontSize':'12px'}}}%%
sequenceDiagram
  participant F as Frame
  participant Y as YOLO + ByteTrack
  participant H as Concealment heuristic
  participant DB as MongoDB
  participant OP as Operator
  participant G as Gemma (local)
  participant E as ElevenLabs

  F->>Y: detect persons + items, track IDs
  Y->>H: item enters torso/hand zone, moves, vanishes
  H->>DB: write candidate event (+ optional alert)
  DB->>OP: surface for review
  Note over OP: a human makes the call
  OP->>G: confirm → draft announcement
  G->>E: announcement text
  E-->>OP: spoken de-escalation audio

Fig. 2 · The live loop. Nothing is announced until a person confirms the event.

The engineering behind it

Four vision stages, not one model

Detection is a pipeline. YOLOv8 finds people and a curated set of carry-able objects; ByteTrack gives each a persistent ID across frames; a ResNet18 feature extractor builds an appearance embedding per track, averaged over a rolling buffer for re-identification; and a ResNet18+LSTM temporal model classifies clip sequences. Layering cheap detection under expensive temporal reasoning keeps the common case fast.

A concealment heuristic built to avoid false flags

Candidate flags come from a geometric heuristic modeled on how people actually handle items. It defines per-person concealment zones (torso, legs) and hand zones, tracks each item's motion, and raises a flag only on a specific sequence: an item that was visible, was handled near a hand, then moved into the torso zone and stayed there or vanished. Requiring movement, persistence, and a plausible body region keeps false positives down. A naive "item overlaps person" rule would fire every time someone holds a product.

Two backends, one detector

A Flask app serves the core detection engine with an in-browser analysis endpoint, and a FastAPI service wraps the same detection function to expose a clean MJPEG stream and write events to MongoDB. The dashboard reads from Mongo, so the vision code and the web app stay decoupled.

Local AI for the live loop, cloud only for review

The announcement is drafted by a local Gemma 4 model via Ollama and voiced by ElevenLabs. This is the piece that won the award, and it means the live loop needs no cloud LLM at all. Cloud vision (Gemini) only ever sees short exported clips, never live streams, and it is opt-in. Analytics run on Snowflake Cortex: events sync from Mongo, and a manager can use Cortex Search for retrieval-grounded questions and Cortex Complete for natural-language-to-SQL over the event history.

Built so a person, not a model, makes the call

Theft-detection tech usually fails people by accusing them automatically. SPOTTED is architected so that can't happen: the system surfaces moments worth a second look, and a human is the required decision-maker for anything that follows.

This posture is structural rather than a policy promise. The pipeline writes candidate events to a queue instead of raising alarms, the announcement path is gated behind an explicit operator confirmation, and the live loop runs entirely on local models, so the sensitive data and the decision both stay in the store.

Two commands to a live pipeline

# Local detector (Flask)
pip install -r requirements.txt   # ultralytics, opencv, torch, Flask, google-genai
python main.py                    # http://localhost:8000

# Streaming backend (FastAPI) + Next.js dashboard
python backend/stream.py          # MJPEG + Mongo writes
cd web && npm install && npm run dev

Optional services activate through environment variables and each degrades gracefully if unset: MONGODB_URI for the event store, GEMMA_API_URL/GEMMA_MODEL for the local announcement model, ELEVENLABS_API_KEY for voice, GEMINI_API_KEY for opt-in clip review, and Snowflake credentials for analytics. The core detect-and-review loop runs with none of them set.

The full loop in about five minutes

Running against the bundled video source, the whole detect, review, and announce loop plays out end to end:

• 0:00The live view shows YOLO boxes and track IDs following people and items, all processed on the local machine.
• 0:45An item moves into a torso zone and the concealment flag fires.
• 1:30The event lands in the dashboard review queue with its clip, waiting on a human decision.
• 2:30Confirming it triggers the announcement: the local Gemma model drafts a de-escalation line and ElevenLabs speaks it aloud.
• 3:30The exported clip can optionally go to Gemini for a second-opinion review.
• 4:15Analytics: a plain-language question about the event history, answered by Snowflake Cortex over the synced data.