Video analytics
Turn camera streams into events, alerts, searchable footage and summaries using real-time detection and tracking plus vision language models, with privacy rules decided before the first camera is connected.
The problem
Many sites have more cameras than staff can watch live. Incidents are found hours later by scrubbing through footage, and simple motion alerts trigger so many false alarms that people stop paying attention.
The engineering problem is to process many streams in real time at a reasonable cost, detect the specific events that matter, cut false positives and let staff search and summarize video in plain language, all under clear rules about what is recorded, who can see it and how long it is kept.
The approach
Current pipelines have two tiers. A real-time tier decodes streams and runs detection, classification and multi-object tracking; DeepStream is NVIDIA's open source, GStreamer-based toolkit for this, with C/C++ and Python interfaces. A reasoning tier uses vision language models to caption clips, answer questions and check alerts. NVIDIA's Video Search and Summarization (VSS) Blueprint in Metropolis combines both tiers and labels each alert as confirmed, rejected or unverified.
Models can be adapted to your cameras with TAO, and pipelines run on edge devices such as Jetson Thor or on data center GPUs, using locally hosted models so video does not leave the site.
For simple counting or line crossing, a smart camera or the analytics built into a video management system may be enough. Custom pipelines pay off when you need site-specific events, natural language search or alert verification across many streams.123
Conceptual architecture
Diagram as a list
Applications & solutions
- Cameras and video management systemProvide live and recorded streamsConnects to Real-time detection and tracking (DeepStream)
- Behavior analytics and rulesTurns tracks into events such as zone entry or proximity incidentsConnects to Alert verification and search agent (VSS)
- Alert verification and search agent (VSS)Verifies alerts and answers questions about footageConnects to Operator console and reports
- Operator console and reportsShows verified alerts, search results and summaries
Models & frameworks
- Vision language model (for example Cosmos Reason)Captions clips and checks whether an alert is realConnects to Alert verification and search agent (VSS)
Inference & runtime software
- Real-time detection and tracking (DeepStream)Detects, classifies and tracks objects across streamsConnects to Event bus (Kafka, Redis Streams or MQTT)
Operations & orchestration
- Event bus (Kafka, Redis Streams or MQTT)Carries detection metadata to downstream servicesConnects to Behavior analytics and rules
Accelerated computing
- Edge or data center GPUsRun the pipeline and modelsConnects to Real-time detection and tracking (DeepStream), Vision language model (for example Cosmos Reason)
Technologies and their roles
metropolis3
Vision AI platform
Collects the models, libraries and blueprints for video analytics agents, including VSS, DeepStream and TAO.
deepstream1
Real-time pipeline
Open source streaming analytics toolkit with hardware-accelerated plug-ins and multi-camera tracking.
blueprints4
Reference application
The VSS Blueprint repository provides a working video search, summarization and alert verification application.
cosmos5
Vision language models
Cosmos reasoning models can produce real-time alerts and dense captions from video.
jetson6
Edge processing
Jetson Thor and other modules run pipelines next to the cameras.
nim2
Model serving
VSS runs its language and vision models as NIM microservices or locally hosted models.
What you need first7
- A documented purpose and legal basis for processing video, with a data protection impact assessment where required
- A camera inventory with stream formats, locations and fields of view
- Labeled examples of the events that matter
- Docker and GPU skills; VSS requires specific Docker, Compose, Container Toolkit and NGC CLI versions
- A validated GPU platform; VSS lists H100, RTX PRO 6000 Blackwell, L40S, DGX Spark, IGX Thor and AGX Thor
Risks and how to reduce them
- Privacy intrusion and surveillance beyond the stated purpose
- Limit the purpose, avoid identifying individuals unless lawfully required, blur faces where possible, keep retention short and log access.
- Bias and misidentification
- Test accuracy across lighting, locations and groups of people, and keep a human in the loop for consequential decisions.
- Services exposed on an open network2
- VSS assumes a trusted, isolated network with authentication, TLS and access control provided by the infrastructure; add these before deployment.
- Alert fatigue
- Use alert verification and tune rules on real footage before going live.
- Evaluation-only license terms8
- VSS microservices use an evaluation license; confirm production terms with NVIDIA.
Related
Sources
- NVIDIA DeepStream SDK (developer page) (opens in a new tab)
- NVIDIA VSS Blueprint documentation: Introduction (opens in a new tab)
- NVIDIA Metropolis for developers (opens in a new tab)
- NVIDIA AI Blueprints GitHub organization (opens in a new tab)
- NVIDIA Cosmos (product page and FAQ) (opens in a new tab)
- NVIDIA Metropolis (product page) (opens in a new tab)
- NVIDIA VSS Blueprint documentation: Prerequisites (opens in a new tab)
- NVIDIA VSS Blueprint documentation: License Information (opens in a new tab)
Thank you. Your correction was sent.
The editors check it against the sources. If you left an email address, they may reply about it.