Smart cities and public spaces
How city governments, transport agencies and venue operators use NVIDIA vision AI, edge computers, video agents, synthetic data and digital twins for traffic, flooding and incident response, with limits on surveillance, biometric identification and data retention set before any camera feed is analyzed.
The problem
City departments often run their own camera, sensor and dispatch systems bought from different vendors, so an event seen by one bureau reaches another late or not at all. In Kaohsiung, flood data held by the water bureau did not flow automatically to the transport bureau, although floods close roads.
Much traffic engineering still rests on manual work: Raleigh's engineers set signal timing from turning counts collected by staff standing at intersections. Detecting a car is easy for older vision systems; understanding an accident, a flooded underpass or a fallen tree is not. Rapid growth adds pressure, and Texas's transport department found its traditional systems could not follow conditions in real time.
Video of public space is also among the most sensitive data a city holds. In the European Union, real-time remote biometric identification by law enforcement in publicly accessible spaces is on the AI Act's list of prohibited practices, as is untargeted scraping of CCTV footage to build facial recognition databases.1234
The approach
Most deployments start with video analytics. Raleigh piloted NVIDIA DeepStream, part of Metropolis, for automated turning counts and reports 95 percent vehicle detection accuracy; Rekor runs Metropolis software and Jetson Xavier NX modules at the roadside in Texas and other states for its traffic management platform.
The newer layer is video agents. NVIDIA's VSS Blueprint, part of Metropolis, pairs vision pipelines with vision language models so operators can ask questions about footage, receive alerts and get written incident reports. Kaohsiung's integrator Linker Vision, Rekor and Raleigh all build on it. For rare events that cameras seldom capture, Cosmos models generate synthetic training video, and Omniverse hosts city-scale twins that show live events in a command center. NVIDIA packages simulation, training and agent deployment as its Blueprint for smart city AI.
A full twin is not always needed. Raleigh's digital twin runs on Esri's ArcGIS on Microsoft Azure, a GIS platform NVIDIA itself calls widely adopted, and loop detectors, radar counters or a vendor's packaged intersection analytics answer many traffic questions. GPU-based agents add value when a city has many feeds, several departments that need the same alerts, and staff to act on them.1235
Conceptual architecture
Diagram as a list
Applications & solutions
- Video search, summarization and alert agents (Metropolis VSS Blueprint)Answer operator questions, raise alerts and draft incident reportsConnects to City digital twin (Omniverse or an existing GIS platform), Operations center and agency dispatch with human confirmation
- City digital twin (Omniverse or an existing GIS platform)Places live events on a 3D map and tests plans before changes on the streetConnects to Operations center and agency dispatch with human confirmation
Models & frameworks
- Synthetic event video and data curation (Cosmos, NeMo Curator)Fill gaps in training data for floods, collapses and other rare eventsConnects to Fine-tuned vision language models on central GPU servers
Inference & runtime software
- Fine-tuned vision language models on central GPU serversDescribe what is happening in a scene, not just which objects appearConnects to Video search, summarization and alert agents (Metropolis VSS Blueprint)
Operations & orchestration
- Operations center and agency dispatch with human confirmationStaff verify alerts and route them to the responsible departments
Accelerated computing
- Roadside and cabinet computers (Jetson modules running DeepStream pipelines)Detect and count objects close to the camera and forward events instead of raw videoConnects to Video search, summarization and alert agents (Metropolis VSS Blueprint)
Networking, power & facilities
- Traffic, flood and public-space cameras and sensorsStream video and readings over the city network to edge or central sitesConnects to Roadside and cabinet computers (Jetson modules running DeepStream pipelines)
Technologies and their roles
NVIDIA Metropolis1
Vision AI platform for city video
Kaohsiung uses Metropolis through Linker Vision's platform, and Rekor uses Metropolis for real-time video understanding on roads in several U.S. states.
NVIDIA DeepStream SDK2
Real-time traffic counting pipelines
Raleigh adopted DeepStream for turning movement counts and NVIDIA reports 95 percent vehicle detection accuracy in the pilot.
NVIDIA Jetson3
Edge computers at the roadside
NVIDIA says Rekor uses Jetson Xavier NX modules for edge AI in Texas, Florida, Philadelphia and other locations.
NVIDIA Blueprints25
Video agent and smart city reference workflows
Kaohsiung, Rekor and Raleigh build agents on the VSS Blueprint, and NVIDIA offers a Blueprint for smart city AI that joins simulation, training and agent deployment.
NVIDIA Omniverse1
City-scale digital twins
Linker Vision builds Kaohsiung's twin in Omniverse from satellite and aerial imagery and links live vision outputs to it.
NVIDIA Cosmos1
Synthetic data and scene description
Linker Vision uses Cosmos to generate video of rare scenarios such as flooding and prompts Cosmos models to describe scenes for responders.
What you need first
- A written policy, approved by elected officials or the relevant oversight body, on what city video may be analyzed for and what is excluded, such as identifying individuals
- An inventory of cameras and sensors with owners, network paths, resolution and retention periods
- Agreement between departments on which events each one must receive and who confirms an alert
- Network capacity and power at edge sites, or a plan to process streams centrally
- Labeled examples of local events, plus a plan for synthetic data where real examples are rare
- A public-facing description of the system, its purpose and how residents can raise concerns
- Procurement terms that keep data ownership and export rights with the city
Risks and how to reduce them
- Function creep from traffic monitoring into surveillance of people
- Limit analytics to vehicles, infrastructure and anonymous counts unless a law and policy allow more, blur faces and plates where not needed, and audit queries made by staff.
- Breach of biometric rules in the European Union and elsewhere4
- Keep real-time remote biometric identification out of scope, and have counsel review every new feature against the AI Act and local law before it goes live.
- False or missed alerts sending crews to the wrong place
- Keep a human confirmation step before dispatch, measure precision and recall per event type, and review missed events after each incident.
- Compromised cameras and edge devices
- Segment camera networks, change default credentials, patch edge devices on a schedule, and encrypt streams between sites.
- Unverified vendor results driving budget decisions
- Ask for baselines and measurement methods before relying on percentage claims, and run a time-boxed pilot with the city's own metrics.
Documented examples
Kaohsiung City Government, with integrator Linker Vision · City government and public infrastructure
Kaohsiung City Government: city video agents and a digital twin built by Linker Vision
Linker Vision built a vision AI platform for Kaohsiung City Government on NVIDIA Metropolis, Omniverse and Cosmos, with video agents that flag events such as roadway flooding and alert the right agencies. No measured results with a stated baseline have been published.
Scaling
Related
Sources
- Linker Vision Taps Into Vision AI to Optimize City Operations (opens in a new tab)
- Raleigh Builds a Smart City With AI and Digital Twins (opens in a new tab)
- Austin Calling: As Texas Absorbs Influx of Residents, Rekor Taps NVIDIA Technology for Roadway Safety, Traffic Relief (opens in a new tab)
- AI Act: regulatory framework for AI (opens in a new tab)
- Build Smart Cities With AI (opens in a new tab)
Thank you. Your correction was sent.
The editors check it against the sources. If you left an email address, they may reply about it.