SPOTTED

What SPOTTED is, in one screen

SPOTTED watches a camera or video source, tracks the people and objects in frame, and surfaces candidate concealment events for a human to review rather than acting on its own. When a reviewer confirms one, the system can generate a spoken de-escalation announcement locally and speak it aloud. Events flow to a database and an analytics warehouse so a manager can ask questions about them in plain language.

An honest note on what ships

The repository is candid that it is a reviewable prototype, not a production surveillance product. The trained theft classifier's weights are intentionally not included, so as shipped both backends run in detection-and-heuristic mode: YOLO finds people and objects, and a geometric concealment heuristic — not a learned theft model — raises the candidate flags. That is a feature of the framing, not a bug hidden by it: the design assumes a person, never the model, makes the call.

The stack, precisely labeled

How the pieces fit together

Heavy pixels stay local. A Python backend does all frame processing on the host and emits only lightweight event records; the Next.js dashboard, the announcement loop, the cloud clip review, and the analytics warehouse all work from those records and exported clips — never a raw video stream.

%%{init: {'theme':'base','themeVariables':{'background':'#0c0c0e','primaryColor':'#151517','primaryTextColor':'#e6e6e6','primaryBorderColor':'#00ff41','lineColor':'#555','secondaryColor':'#111','tertiaryColor':'#0a0a0a','clusterBkg':'#0c0c0e','clusterBorder':'#262626','fontFamily':'JetBrains Mono, monospace','fontSize':'13px','textColor':'#c0c0c0'}}}%%
flowchart TB
  subgraph EDGE["Local host (heavy pixels stay here)"]
    CAM["Camera / video"]
    CV["YOLOv8 + ByteTrack
ResNet re-ID + concealment heuristic"]
  end
  subgraph BACK["Backends"]
    ST["FastAPI stream
MJPEG + event writes"]
  end
  subgraph STORE["Event store"]
    MG[("MongoDB
events · alerts · cameras")]
  end
  subgraph WEB["Next.js 16 dashboard (R3F)"]
    OPS["Operator review · alerts · events"]
    ANA["Analytics chat"]
  end
  subgraph LOOP["Announce loop"]
    GEM["Gemma 4 (Ollama)
draft announcement"]
    EL["ElevenLabs TTS
speak deterrent"]
  end
  subgraph CLOUD["Cloud (candidate clips + analytics)"]
    GEMINI["Gemini 2.5
clip verdict (opt-in)"]
    SNOW[("Snowflake Cortex
Search + Complete")]
  end
  CAM --> CV --> ST --> MG
  MG --> OPS
  OPS -->|"confirm"| GEM --> EL
  OPS -->|"export clip"| GEMINI
  MG -->|"sync"| SNOW --> ANA

Fig. 1 — Component architecture. Only event records and exported clips ever leave the host.

The detection-to-deterrent loop

The live path is a chain of models feeding a heuristic, gated by a human before anything is spoken.

%%{init: {'theme':'base','themeVariables':{'background':'#0c0c0e','primaryColor':'#151517','primaryTextColor':'#e6e6e6','primaryBorderColor':'#00ff41','lineColor':'#555','actorBkg':'#151517','actorBorder':'#00ff41','actorTextColor':'#e6e6e6','signalColor':'#666','signalTextColor':'#c0c0c0','noteBkgColor':'#111','noteTextColor':'#c0c0c0','noteBorderColor':'#333','fontFamily':'JetBrains Mono, monospace','fontSize':'12px'}}}%%
sequenceDiagram
  participant F as Frame
  participant Y as YOLO + ByteTrack
  participant H as Concealment heuristic
  participant DB as MongoDB
  participant OP as Operator
  participant G as Gemma (local)
  participant E as ElevenLabs

  F->>Y: detect persons + items, track IDs
  Y->>H: item enters torso/hand zone, moves, vanishes
  H->>DB: write candidate event (+ optional alert)
  DB->>OP: surface for review
  Note over OP: human decides — no auto-accusation
  OP->>G: confirm → draft announcement
  G->>E: announcement text
  E-->>OP: spoken de-escalation audio

Fig. 2 — Live loop. The human review step (highlighted) is the design's safety keystone.

Design decisions & the trade-offs behind them

Four vision stages, not one model

Detection is a pipeline. YOLOv8 finds people and a curated set of carry-able objects; ByteTrack gives each a persistent ID across frames; a ResNet18 feature extractor builds an appearance embedding per track, averaged over a rolling buffer for re-identification; and a ResNet18+LSTM temporal model classifies clip sequences as normal or theft when its weights are present. Layering cheap detection under expensive temporal reasoning keeps the common case fast.

The concealment heuristic — the real shipped logic

Because the learned classifier's weights aren't in the repo, the flags you actually see come from a geometric heuristic, and it is genuinely considered engineering. It defines per-person concealment zones (torso, legs) and hand zones, tracks each item's motion and displacement, and raises a candidate only on a specific transition: an item that was visible outside the body, interacted with near a hand, then moved into the torso zone and stayed — or vanished near the torso while last seen in-hand. It deliberately requires movement, persistence, and a plausible body region to fire, precisely because a naive "item overlaps person" rule would be a false-positive machine.

Two backends, one detector

The original detector is a Flask app with an in-browser analysis endpoint; a second FastAPI service wraps the same detection function to expose a clean MJPEG stream and write events to MongoDB. It is honest to note this duplication — the temporal model class is defined in three places — and to treat consolidating on one service as future work.

Local-first AI, cloud only for review

The announcement is drafted by a local Gemma 4 model via Ollama and voiced by ElevenLabs — the award-winning piece, and the reason the live loop needs no cloud LLM. Cloud vision (Gemini) is reserved for reviewing exported candidate clips, never live streams, and it is opt-in. Analytics run on Snowflake Cortex: events sync from Mongo, and a manager can use Cortex Search for RAG-style questions and Cortex Complete for natural-language-to-SQL over the event history.

The real risk is being wrong about a person

Security threat-modeling matters here, but it is secondary. The defining risk of a system that watches people and outputs "theft" is a false accusation of a real human being — and the way that risk distributes unequally across who gets watched and flagged. A senior review will probe this first, so it belongs first.

%%{init: {'theme':'base','themeVariables':{'background':'#0c0c0e','primaryColor':'#151517','primaryTextColor':'#e6e6e6','primaryBorderColor':'#00ff41','lineColor':'#555','clusterBkg':'#0c0c0e','clusterBorder':'#262626','fontFamily':'JetBrains Mono, monospace','fontSize':'12px','textColor':'#c0c0c0'}}}%%
flowchart LR
  subgraph Z1["Subjects (no consent flow)"]
    P["People in frame"]
  end
  subgraph Z2["Local host"]
    CV["CV pipeline"]
    BIO[("Track snapshots
crops + embeddings")]
  end
  subgraph Z3["Decision"]
    HUM["Human reviewer
(required gate)"]
  end
  subgraph Z4["Downstream"]
    MG[("MongoDB")]
    CLOUD["Gemini clip / Snowflake"]
    ACT["Announcement / action"]
  end
  P ==>|"TB①  surveillance, no consent"| CV
  CV --> BIO
  CV --> MG --> HUM
  HUM ==>|"TB②  the accountability gate"| ACT
  MG -.->|"TB③  clips + records to cloud"| CLOUD

Fig. 3 — Trust boundaries. TB② — the human gate — is what separates a review tool from an automated accuser.

Ethics & privacy

Security

On the classic axis: the MJPEG stream binds all interfaces with wildcard CORS and no authentication, so an open video feed of a monitored space is a real exposure; a MongoDB connection string (host + user) sits in the committed env examples; and the biometric snapshots are unencrypted on disk. The fixes are conventional — authenticate the stream, bind loopback or put it behind a gateway, move secrets out of the repo, and encrypt the snapshot store.

Running it

# Local detector (Flask) — YOLO-only if video_model.pth is absent
pip install -r requirements.txt   # ultralytics, opencv, torch, Flask, google-genai
python main.py                    # http://localhost:8000

# Streaming backend (FastAPI) + Next.js dashboard
python backend/stream.py          # MJPEG + Mongo writes
cd web && npm install && npm run dev

Optional services activate through environment variables and each degrades gracefully if unset: MONGODB_URI for the event store, GEMMA_API_URL/GEMMA_MODEL for the local announcement model, ELEVENLABS_API_KEY for voice, GEMINI_API_KEY for opt-in clip review, and Snowflake credentials for analytics. Without the trained video_model.pth, both backends run detection-and-heuristic mode.

What verification looks like for a CV system

Detection quality here is validated visually and by labeled-video playback — the evaluation script runs the classifier over folders of shoplift and normal clips with ground-truth overlays. That is the right instinct, but for a system whose errors land on people, the strategy has to be more formal than eyeballing.

The strategy to articulate

The honest plan has three tiers. First, a labeled benchmark with real precision/recall and, critically, a false-positive rate for the concealment heuristic — the metric that matters most when a false flag means a wrongful stop. Second, unit tests around the deterministic geometry (zone membership, track transitions, IoU) that drives the flags. Third, and non-negotiable before the learned classifier ships, a fairness evaluation of flag rates across demographic groups. Being able to say that the current evidence is qualitative and to name the metrics you'd hold it to is the senior move.

A five-minute live walkthrough

Run against the bundled video source so behavior is deterministic. Lead with the pipeline, land on the human gate and the local announcement — the two things that make it defensible and the thing that won the award.

• 0:00Open the live view. Point out YOLO boxes + track IDs following people and items. "Detection and tracking are running locally on the host — no video leaves this machine."
• 0:45Walk an item into a torso zone. Show the candidate concealment flag fire, and stress the wording: "candidate, not verdict."
• 1:30Switch to the dashboard — the event appears in the operator review queue with its clip. "A person reviews this. The system never accuses on its own."
• 2:30Confirm the event and trigger the announcement: a local Gemma model drafts a de-escalation line, ElevenLabs speaks it. "This is the Best-Use-of-Gemma piece — and it runs locally."
• 3:30Show optional Gemini clip review on the exported clip. "Cloud vision only ever sees a short candidate clip, never the stream — that's the privacy line."
• 4:15Open analytics: ask a plain-language question and show Cortex answering over synced event data. Close on §4 — "the design's real feature is the human in the loop."

Ninety seconds, no jargon

The technical explanation

The questions this project invites

Grouped by where the pressure comes from. The hardest questions here are ethical, and the answers meet them head-on — a serious panel is testing whether you saw the risk before they had to raise it.

On responsibility

On the vision engineering

On architecture

The honest roadmap

Known limitations, tracked as findings

Documented like any other assessment — each limitation logged as a severity-tagged finding with a fix. For a system that watches people, the top findings are as much about ethics as engineering.