Caterpillar: an on-machine AI assistant on NVIDIA Jetson Thor for construction equipment

Construction machines work on sites that change daily, often with weak connectivity, and operators have to manage settings, hazards and troubleshooting while keeping their hands on the controls. Caterpillar says it wants machines that turn sensor data into recommendations at the edge, with in-cab coaching, productivity tips and safety alerts, and that are prepared for AI-assisted and possibly autonomous work.

Operator guidance also has to draw on accurate machine information, which Caterpillar keeps in its Helios data platform, rather than on general web knowledge.¹

Cat AI Assistant on the machine

Caterpillar debuted the Cat AI Assistant at CES in January 2026. In the live demo, NVIDIA's blog says, the assistant ran inside a Cat 306 CR mini excavator on an NVIDIA Jetson Thor module: speech recognition and synthesis used NVIDIA Riva with Parakeet and Magpie models (NVIDIA attributes these speech models to Nemotron), a Qwen3 4B language model served with vLLM interpreted requests, and Helios supplied machine context. NVIDIA writes that the language model ran with low latency and no cloud connection. Operators used voice commands to set range-of-motion limits, such as a floor above known underground utilities, under a feature NVIDIA calls E-Ceiling, and asked the assistant about controls and machine behavior.

Factory twins

Caterpillar says it is building digital twins of its factories with NVIDIA Omniverse libraries and OpenUSD to test layouts and processes before changing them. NVIDIA describes this as a pilot covering several US sites that build construction products and large mining machines.

Status

The assistant was a CES demonstration, and NVIDIA presents the in-cab features as previews while the companies explore extending them across fleets and jobsites. Caterpillar's release gives no availability dates.¹²

What others can learn: a speech front end, a small language model and the maker's own machine data can run together on an edge module inside a cab, so guidance keeps working where coverage is poor, and voice-set digging limits show how an assistant can support existing safety functions rather than replace them. Limits: the sources describe a single trade show demo and a factory pilot, with no published field results, availability dates or safety approvals for the in-cab features. Contractors who mainly need equipment telematics, manuals and fault codes can already get them from dealer portals and fleet software, and a cloud chatbot may be enough where sites have reliable connectivity.