IntelligentHQ
IntelligentHQ
Skip to content

AI infrastructure and compute

The compute under AI workloads on Azure: GPU virtual machines in the ND, NC, NV and NG families, container platforms (Azure Kubernetes Service and Azure Container Apps), and Microsoft's own Maia accelerators that power Microsoft services.

4 technology profiles in this area

In this area

Overview

Azure's GPU virtual machines come in families. ND targets large-memory, compute-intensive work and includes GB300, GB200, MI300X, H200 and H100 series; NC covers compute and visualization; NV and NG target virtual desktops, graphics and cloud gaming.

Most teams run AI services in containers. AKS Automatic is Microsoft's recommended mode for most new Kubernetes workloads. Azure Container Apps hides the Kubernetes API, can scale to zero and is where many teams start. Microsoft's agent guidance suggests Container Apps with serverless GPU for small language models and AKS for large models on large GPU clusters.

Microsoft's Maia 200 accelerator is in production serving Microsoft workloads, including Microsoft Copilot, with Foundry and GitHub Copilot to follow. Microsoft's announcement does not say customers can rent Maia directly.

If you only call hosted models, you may never manage GPUs at all: running your own GPU containers makes sense mainly when you want to run your own models rather than use Foundry endpoints.12345

Problems it addresses

  • Matching GPUs to the job1

    Microsoft groups GPU sizes by workload: ND for large-memory compute, NC for compute and visualization, NV and NG for desktops and graphics.

  • Kubernetes overhead2

    AKS Automatic handles node provisioning, scaling, upgrades and security defaults, and adds a pod readiness SLA.

  • Bursty inference traffic3

    Container Apps scales on traffic or event sources such as queues, including to zero, without exposing Kubernetes APIs.

  • Paying for the control plane2

    AKS creates and manages the Kubernetes control plane at no cost; you pay for the nodes that run your applications.

A typical workflow

  1. Decide whether to self-host models4

    Use Foundry endpoints unless you need to run your own model on your own GPUs.

  2. Pick the container platform4

    Choose Container Apps for serverless microservices and small models, AKS for full Kubernetes control and large GPU clusters.

  3. Select VM sizes1

    Match the ND, NC, NV or NG family to training, inference or visualization needs.

  4. Run agents on CPU4

    Run the agent itself on CPU-based AKS or Container Apps while the model runs on GPUs.

  5. Observe the cluster2

    Use managed Prometheus, Container Insights and Grafana dashboards, which are on by default in AKS Automatic.

Technologies

  • Cloud service

    Azure Container Apps

    Azure Container Apps runs containers as a serverless service: you deploy images, set scale rules and ingress, and Azure runs the underlying Kubernetes, Dapr, KEDA and Envoy for you. Apps can scale to zero, run jobs, use serverless GPUs, and give AI agents sandboxed code execution through dynamic sessions.

    • Infrastructure
    • Developer
    • AI
  • Hardware family

    Azure Maia and Azure Cobalt

    Azure Maia and Azure Cobalt are Microsoft-designed chips in Azure datacenters. Maia accelerators run AI inference for Microsoft services, with no direct customer rental stated. Cobalt Arm CPUs power VM sizes customers can deploy: Cobalt 100 sizes such as Dpsv6, and Cobalt 200 sizes in early access preview.

    • Infrastructure
  • Cloud service

    Azure GPU virtual machines

    Azure's GPU-accelerated virtual machines are the N-series VM families: ND for large-scale AI training and inference, NC for compute-intensive AI and HPC, and NV and NG for virtual desktops, graphics and cloud gaming. They use NVIDIA or AMD GPUs and are billed like other Azure VMs.

    • Infrastructure
    • AI
  • Cloud service

    Azure Kubernetes Service (AKS)

    Azure Kubernetes Service is Microsoft's managed Kubernetes. Azure runs the control plane at no charge, you pay for the worker nodes, and you choose between AKS Automatic (Azure manages nodes, scaling, upgrades and security defaults) and AKS Standard (you configure and run the node pools).

    • Infrastructure
    • Developer

Use cases

Next steps

  1. Write down model size, expected requests per second and latency target before picking a VM family.

  2. Start new Kubernetes clusters in AKS Automatic unless you need Windows node pools, custom networking or specific VM SKUs.

  3. Check GPU series availability in your target regions before committing to a design.

  4. Compare running your own model with calling a Foundry endpoint on total cost, not only GPU price.

Connected areas

Sources

  1. Virtual machine sizes overview (opens in a new tab)Microsoft · Vendor-reported
  2. What is Azure Kubernetes Service (AKS)? (opens in a new tab)Microsoft · Vendor-reported
  3. Comparing Container Apps with other Azure container options (opens in a new tab)Microsoft · Vendor-reported
  4. Technology plan for AI agents (Cloud Adoption Framework) (opens in a new tab)Microsoft · Vendor-reported
  5. Microsoft to expand Azure Maia system to multi-gigawatt deployments (opens in a new tab)Microsoft · Vendor-reported

Fill out the form below to request your copy.

Name(Required)