Version

Scale the platform

On this page

This document is for an unreleased version of Modelplane.

This document applies to the Modelplane main branch and not to the latest release v0.6.

You have one small-GPU cluster with a running model. In this guide, you’ll grow the fleet with larger-GPU capacity so the ML team has more to schedule against.

Provisioning takes about 10 to 15 minutes.

Register more clusters

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l40s-1x-g6e
spec:
  description: "EKS g6e.xlarge, 1x NVIDIA L40S"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6e.xlarge
      diskSizeGb: 100
      accelerator:
        type: nvidia-l40s
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "46068Mi" }
---
# g6e.xlarge is available in us-east-1, us-west-2, and eu-central-1.
# eu-west-1 does NOT have g6e.xlarge.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-us-west
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: EKS
    eks:
      region: us-west-2
  nodePools:
  - name: gpu-l40s
    className: l40s-1x-g6e
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-west-2a
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-eu-central
  labels:
    modelplane.ai/region: eu-central
spec:
  cluster:
    source: EKS
    eks:
      region: eu-central-1
  nodePools:
  - name: gpu-l40s
    className: l40s-1x-g6e
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - eu-central-1a

Note

g6e.xlarge runs ~$2/hr on demand. Two of them plus the L4 from earlier is a few dollars for this tour. Clean up when you’re done (see Clean up ).

Register two more clusters with a bigger hardware class: A100 (40 GB) in us-west and us-east. Apply the manifest:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-a100-40-1x
spec:
  description: "GKE a2-highgpu-1g, 1x NVIDIA A100 40GB"
  provisioning:
    provider: GKE
    gke:
      machineType: a2-highgpu-1g
      diskSizeGb: 200
      accelerator:
        type: nvidia-tesla-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      # A100 40GB real reported VRAM. Keep the selector at >= 35Gi (not >= 40Gi)
      # so it reliably clears the L4 (24Gi) without hitting the boundary.
      memory: { value: "40960Mi" }
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-us-west
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: GKE
    gke:
      region: us-west1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-west1-b
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-us-east
  labels:
    modelplane.ai/region: us-east
spec:
  cluster:
    source: GKE
    gke:
      region: us-east1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-east1-b

kubectl apply -f https://docs.modelplane.ai/main/examples/getting-started/gke/platform-scale.yaml

Note

a2-highgpu-1g runs ~$3.50/hr on demand. Two of them plus the L4 from earlier is a few dollars for this tour. Clean up when you’re done (see Clean up ).

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a100-1x
spec:
  description: "AKS Standard_NC24ads_A100_v4, 1x NVIDIA A100 80GB"
  provisioning:
    provider: AKS
    aks:
      vmSize: Standard_NC24ads_A100_v4
      diskSizeGb: 100
      accelerator:
        type: nvidia-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      memory: { value: "81920Mi" }
---
# verify: Standard_NC24ads_A100_v4 quota/availability in eastus and southcentralus.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-eastus
  labels:
    modelplane.ai/region: eastus
spec:
  cluster:
    source: AKS
    aks:
      location: eastus
  nodePools:
  - name: gpua100
    className: a100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-southcentralus
  labels:
    modelplane.ai/region: southcentralus
spec:
  cluster:
    source: AKS
    aks:
      location: southcentralus
  nodePools:
  - name: gpua100
    className: a100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Note

Standard_NC24ads_A100_v4 runs ~$3.70/hr on demand. Two of them plus the A10 from earlier is a few dollars for this tour. Clean up when you’re done (see Clean up ).

Nebius projects are bound to one region, so you grow the fleet by GPU tier rather than geography. Register a bigger H100 (80 GB) cluster in the same region:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: h100-1x
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 100
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      memory: { value: "81559Mi" }   # H100's real reported VRAM (not the nominal 80GB)
---
# Nebius projects are bound to one region, so this cluster runs in the same
# region as the base cluster. The fleet scales by GPU tier (L40S -> H100), not
# geography. To add a region, create another project + ClusterProviderConfig.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north-h100
  labels:
    modelplane.ai/region: eu-north
    modelplane.ai/tier: h100
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-h100
    className: h100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Note

The H100 cluster costs more per hour than the L40S from earlier. Clean up when you’re done (see Clean up ).

Vultr GPU plans are region-gated: ewr offers only A16 plans, and the A40 is sold in lax and syd. Register a cluster with the bigger hardware class, A40 (48 GB), in lax:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a40-1x
spec:
  description: "Vultr vcg-a40-24c-120g-48vram, 1x NVIDIA A40 48GB"
  provisioning:
    provider: Vultr
    vultr:
      plan: vcg-a40-24c-120g-48vram
      accelerator:
        type: nvidia-a40
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.6.0" }
    capacity:
      memory: { value: "46068Mi" }   # The A40's real reported VRAM (not the nominal 48GB)
---
# Vultr GPU plans are region-gated: the A40 plan is offered in lax and syd,
# while the A16 regions carry only A16 plans.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: vultr-lax
  labels:
    modelplane.ai/region: lax
spec:
  cluster:
    source: Vultr
    vultr:
      region: lax
      # VKE accepts only the exact versions it currently offers; list them
      # with vultr-cli kubernetes versions.
      kubernetesVersion: "v1.36.6+1"
  nodePools:
  - name: gpu-a40
    className: a40-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Note

vcg-a40-24c-120g-48vram runs ~$1.70/hr on demand. It plus the A16 from earlier is a few dollars for this tour. Clean up when you’re done (see Clean up ).

Civo GPU sizes are region-gated: NYC1 offers L40S sizes, and LON1 offers the A100 and H100. Register a cluster with the bigger hardware class, A100 (80 GB), in LON1:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a100-1x
spec:
  # Civo GPU sizes are region-gated: NYC1 offers L40S sizes while LON1
  # offers A100 and H100. List a region's sizes with the
  # /v2/sizes?region=<region> API before picking another size.
  description: "Civo an.g1.a100sxm.kube.x1, 1x NVIDIA A100 SXM 80GB"
  provisioning:
    provider: Civo
    civo:
      size: an.g1.a100sxm.kube.x1
      accelerator:
        type: nvidia-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      memory: { value: "81920Mi" }
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: civo-lon1
  labels:
    modelplane.ai/region: lon1
spec:
  cluster:
    source: Civo
    civo:
      region: LON1
  nodePools:
  - name: gpu-a100
    className: a100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Note

The A100 cluster costs more per hour than the L40S from earlier. Clean up when you’re done (see Clean up ).

Modelplane provisions the new clusters in parallel:

kubectl wait --for=condition=Ready ic --all --timeout=20m

Your model keeps running

Growing the fleet doesn’t disturb anything already deployed. qwen-demo stays on its original cluster and the new capacity comes online the moment it’s Ready with no interruption for the ML team. A replica only moves if its deployment changes in a way that no longer fits where it runs.

Next step

The fleet has grown with larger-GPU capacity. The ML team is next. Scale the model to serve it across the fleet behind one endpoint.