Architecture, engineering and construction
How architects, engineers, contractors and equipment makers use AI and NVIDIA technologies: jobsite and building digital twins on OpenUSD, synthetic images for site safety models, camera agents that watch for hazards, and assistants inside construction machines, plus when BIM tools already cover the need.
The problem
Every building or civil project is a one-off assembled by many firms. Architects, structural and services engineers, contractors and equipment suppliers each model their part in different software, so coordination depends on exporting and merging files. When merging is slow, teams update less often, and clashes or sequencing mistakes surface on site, where they cost the most.
Jobsites are also among the most dangerous workplaces. Cranes, excavators and people share a space that changes every day, buried utilities and overhead lines are easy to hit, and the rare events that cause serious accidents are exactly the ones for which there are few photos to train a detection model.
Machines and crews work far from offices, often with poor connectivity, and skilled operators are hard to find. Operators need help with settings, limits and troubleshooting at the controls, not in a manual back at the site office.1
The approach
Digital twins come first. NVIDIA's AEC page names Omniverse for twins in building design, civil engineering, smart cities and facility operations. Bouygues Construction, according to NVIDIA, built its own twin applications on Omniverse and OpenUSD that merge Revit, SolidWorks and Cesium data in one workspace, simulate phasing, clearances and heavy lifts, and use Isaac Sim to generate training images of rare or hazardous site situations. Bechtel announced in October 2025 that it is turning NVIDIA's Omniverse DSX blueprint for gigawatt-scale AI factories into a modular design it can repeat across projects.
On site, NVIDIA says Shimizu Corporation is piloting the Metropolis Video Search and Summarization blueprint to improve construction worker safety. Inside the cab, Caterpillar's CES 2026 demo put a spoken assistant on a Jetson Thor module in a small excavator; its speech recognition and voice came from models NVIDIA credits to Nemotron, it needed no network connection, and the operator used it to cap digging depth over buried utilities.
Many needs are met without these tools. Clash detection, 4D sequencing and model federation are standard features of mainstream BIM and coordination platforms, reality capture services produce point clouds without custom AI, and site cameras with vendor analytics handle routine perimeter and PPE checks. A custom twin, synthetic data or on-machine AI earns its cost on large or repeated programs, on sites where rare hazards matter most, or where machines must work without coverage.12345
Conceptual architecture
Diagram as a list
Applications & solutions
- BIM, CAD, terrain and schedule dataSupplies building, equipment and site geometry plus the construction sequenceConnects to Jobsite or building twin (Omniverse on OpenUSD)
- Jobsite or building twin (Omniverse on OpenUSD)Merges discipline models to test phasing, clearances and liftsConnects to Synthetic site imagery (Isaac Sim), Handover twin for owners and operators
- Site management, safety reporting and fleet dataRecords incidents, progress and machine data and feeds them back to the twinConnects to Jobsite or building twin (Omniverse on OpenUSD)
- Handover twin for owners and operatorsDelivers the as-built model in OpenUSD and returns owner changes for later phasesConnects to Jobsite or building twin (Omniverse on OpenUSD)
Models & frameworks
- Synthetic site imagery (Isaac Sim)Renders rare or hazardous scenes to train detection modelsConnects to Site camera agents (Metropolis VSS blueprint)
Inference & runtime software
- Site camera agents (Metropolis VSS blueprint)Search, summarize and flag safety events in site videoConnects to Site management, safety reporting and fleet data
- On-machine assistant (Jetson Thor, speech models)Answers operators and sets motion limits on the machine without a cloud linkConnects to Site management, safety reporting and fleet data
Technologies and their roles
NVIDIA Omniverse1
Jobsite and building twins
NVIDIA's AEC page names Omniverse for design and civil engineering twins, and Bouygues Construction built its twin applications on it.
NVIDIA Isaac1
Synthetic data for site safety models
Bouygues Construction uses Isaac Sim to generate training images of rare or hazardous jobsite situations, per NVIDIA.
NVIDIA Metropolis4
Video agents for worker safety
NVIDIA says Shimizu Corporation is piloting the Metropolis VSS blueprint to improve construction worker safety.
NVIDIA Jetson6
AI computer inside construction machines
Caterpillar's CES 2026 demo ran the Cat AI Assistant on a Jetson Thor module fitted to a Cat 306 CR, and Caterpillar says Thor brings real-time inference to its equipment.
NVIDIA Nemotron5
Speech models for operator assistants
NVIDIA attributes the speech recognition and synthesis models in Caterpillar's in-cab assistant to Nemotron.
What you need first
- Coordinated BIM and CAD models with agreed exchange formats, and a person responsible for keeping the twin current
- Survey, terrain and reality capture data for the site
- Clear ownership and licensing terms for design data shared among owner, designers and contractors
- A site safety plan that defines which events cameras should flag and who responds
- Agreement with workers and their representatives on camera placement, purpose and retention
- Machine data access from equipment makers or telematics providers if on-machine or fleet AI is planned
- GPU workstations or servers for simulation, checked against each tool's published requirements
Risks and how to reduce them
- Surveillance of site workers
- Point cameras at zones and equipment rather than individuals, blur faces where events allow, publish the purpose and retention period, and follow local data protection and labor rules.
- Over-reliance on AI around heavy equipment
- Treat assistants and camera alerts as aids; certified machine safety systems, exclusion zones, spotters and permit procedures stay in place, and utility locations are still confirmed on site.
- Confidential design data in shared twins
- Client and supplier models often carry confidentiality and copyright terms; control access per layer and confirm rights before a model is shared or handed over.
- Results drawn from demos and single firms
- The published examples are a trade show demo, pilots and vendor-written stories; measure a baseline on your own project and pilot one use before rolling out.
Documented examples
Caterpillar · Construction and mining equipment
Caterpillar: an on-machine AI assistant on NVIDIA Jetson Thor for construction equipment
At CES 2026 Caterpillar demonstrated its Cat AI Assistant running on NVIDIA Jetson Thor inside a Cat 306 CR mini excavator, answering spoken requests and setting digging limits without a cloud link. Caterpillar is also piloting factory digital twins on Omniverse libraries. No jobsite rollout date has been published.
Demonstration only
Related
Sources
- Digital twins power sustainable building innovation: Bouygues Construction (customer story) (opens in a new tab)
- NVIDIA architecture, engineering and construction (industry page) (opens in a new tab)
- Bechtel to accelerate AI data center construction with NVIDIA (opens in a new tab)
- NVIDIA and Japan bring full-stack AI and robotics to every industry (opens in a new tab)
- Steel, sensors and silicon: how Caterpillar is bringing edge AI to the jobsite (opens in a new tab)
- Caterpillar and NVIDIA expand collaboration on physical AI and robotics (Caterpillar press release) (opens in a new tab)
Thank you. Your correction was sent.
The editors check it against the sources. If you left an email address, they may reply about it.