Try Modelplane locally

Modelplane splits AI inference between two teams. The platform or inference team publishes the GPU capacity in its fleet, and the ML team declares what each model needs. Modelplane matches the two, so neither team waits on the other. The platform team can add or replace clusters without touching a deployment, and the ML team can change models without filing a request.

This demo walks you through the Modelplane workflow on two kind clusters on your machine. A setup script creates the clusters. Then you build the inference platform like a platform team, and upgrade a model behind the same name like an ML team.

Prerequisites

Before you begin, make sure you have:

• Docker, with at least 10 GB of memory on Docker Desktop. Both clusters and the model run on your machine.
• kind installed
• kubectl installed
• Helm installed
• jq installed

Set up the clusters

Run the setup script. It takes about 10 minutes:

curl -fsSL https://docs.modelplane.ai/main/examples/getting-started/local-demo/setup.sh | bash

The script creates two kind clusters:

• modelplane, the control plane. It runs Crossplane and Modelplane, and holds every Modelplane object in this demo.
• modelplane-workload, the workload cluster. It replaces the GPU cluster you’d run in a cloud account. MetalLB gives it load balancer addresses, and dra-example-driver gives it eight fake GPUs.

It also stores the workload cluster’s kubeconfig in a Secret on the control plane. When the script finishes, kubectl points at the control plane.

Build the platform

The platform team describes its fleet to Modelplane. It publishes hardware profiles and registers the clusters that offer them. It also runs the gateway that callers send requests to. The platform team writes these objects without needing to know which models will run.

Apply the platform manifest:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: synthetic-gpu
spec:
  description: Fake GPU from dra-example-driver
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.example.com
    deviceClassName: gpu.example.com
    count: 1
    capacity:
      memory: { value: "80Gi" }
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: local
  labels:
    modelplane.ai/region: local
spec:
  cluster:
    source: Existing
    existing:
      secretRef:
        name: local-cluster-kubeconfig
        key: kubeconfig
  nodePools:
  - name: gpu-synthetic
    className: synthetic-gpu
    # Pretend the cluster has two nodes, one per model. It really has one.
    nodeCount: 2
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceGateway
metadata:
  name: public
spec:
  clusterName: local
---
apiVersion: v1
kind: Namespace
metadata:
  name: ml-team

The InferenceClass publishes a hardware profile: one GPU with 80Gi of memory. ML teams match their models against classes, so they can choose hardware without knowing which clusters exist.

The InferenceCluster registers the workload cluster through the kubeconfig the script stored. It offers a node pool with synthetic-gpu hardware. Modelplane installs its serving stack on the cluster, including Envoy Gateway and llm-d. This demo registers one cluster, but Modelplane is built to manage a fleet of them, and the platform team can add more without changing any model.

The InferenceGateway gives callers one address, however many clusters serve the models behind it. It accepts OpenAI API requests and routes each one to a cluster serving the model it names.

The ml-team namespace holds the ML team’s objects, mainly the models you deploy in the next section.

Wait for Modelplane to install the serving stack and bring up the gateway:

kubectl wait ic/local --for=condition=Ready --timeout=15m
kubectl wait ig/public --for=condition=Ready --timeout=10m

The gateway’s address is on kind’s Docker network, which your machine can’t reach on macOS or Windows. Forward localhost:8080 to it:

curl -fsSL https://docs.modelplane.ai/main/examples/getting-started/local-demo/setup.sh | bash -s -- --expose

Deploy a model

A ModelDeployment describes a model and the hardware it needs. A ModelService gives one or more deployments a name that callers use, in the form <namespace>/<service>. Applications call the name, and the team can change the models behind it.

The ML team wants an assistant that answers technical questions. Start with SmolLM-135M-Instruct, the first generation of Hugging Face’s SmolLM models, and name it ml-team/assistant:

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: smollm
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: smollm
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: device.capacity["gpu.example.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: ghcr.io/ggml-org/llama.cpp:server@sha256:991cf50e9eb7dee4c18090849c7f909bafc5a1884cdde2dc3011df7407da09d6
                env:
                - { name: LLAMA_ARG_HF_REPO, value: "HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF" }
                - { name: LLAMA_ARG_HF_FILE, value: "smollm-135m-instruct-add-basics-q8_0.gguf" }
                - { name: LLAMA_ARG_ALIAS, value: "$(MODELPLANE_SERVED_MODEL_NAME)" }
                - { name: LLAMA_ARG_HOST, value: "0.0.0.0" }
                - { name: LLAMA_ARG_PORT, value: "8000" }
                - { name: LLAMA_ARG_CTX_SIZE, value: "2048" }
---
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: assistant
  namespace: ml-team
spec:
  endpoints:
  - name: smollm
    selector:
      matchLabels:
        modelplane.ai/deployment: smollm

Wait for the deployment to report Ready, and for the gateway to route to it:

kubectl -n ml-team wait md/smollm --for=condition=Ready --timeout=10m
kubectl -n ml-team wait modelservice/assistant --for=condition=RoutingReady --timeout=10m

The nodeSelector asks for one GPU with at least 20Gi of memory, and names no cluster. It’s a Kubernetes Dynamic Resource Allocation (DRA) request, the same one you’d write for a pod. Modelplane compares it with the classes the platform team published, finds that synthetic-gpu fits, and places a replica on a cluster that offers it. See where the replica went:

kubectl -n ml-team get modelreplica

The CLUSTER column shows local. On a real fleet, the platform team can add, replace or move clusters, and the deployment stays the same.

llama.cpp runs the model on the workload cluster’s CPU. The fake GPU is a DRA device that gives the scheduler something to claim.

Call the model

The gateway answers on localhost:8080. Send it a request in the OpenAI format:

curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model": "ml-team/assistant", "messages": [{"role": "user", "content": "What is Kubernetes?"}]}' \
  | jq -r '.model, .choices[0].message.content'

The model field in the response shows which deployment answered, ml-team/smollm. Your request asked for one model name, ml-team/assistant. From that name, the gateway found the smollm endpoint and forwarded the request. Callers don’t hold a cluster, address or engine name, which leaves the team free to replace the model.

Swap in a better model

Read the answer. For better ones, the team turns to SmolLM2-360M-Instruct. Hugging Face trained SmolLM2 on more data, and this model has close to three times the parameters.

A ModelService can list several endpoints, each with a priority. Requests go to the priority 0 endpoint while it has a ready replica, and to the priority 1 endpoint when it has none.

Deploy SmolLM2-360M-Instruct, put it in assistant at priority 0, and move the first model to priority 1:

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: smollm2
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: smollm2
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: device.capacity["gpu.example.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: ghcr.io/ggml-org/llama.cpp:server@sha256:991cf50e9eb7dee4c18090849c7f909bafc5a1884cdde2dc3011df7407da09d6
                env:
                - { name: LLAMA_ARG_HF_REPO, value: "HuggingFaceTB/SmolLM2-360M-Instruct-GGUF" }
                - { name: LLAMA_ARG_HF_FILE, value: "smollm2-360m-instruct-q8_0.gguf" }
                - { name: LLAMA_ARG_ALIAS, value: "$(MODELPLANE_SERVED_MODEL_NAME)" }
                - { name: LLAMA_ARG_HOST, value: "0.0.0.0" }
                - { name: LLAMA_ARG_PORT, value: "8000" }
                - { name: LLAMA_ARG_CTX_SIZE, value: "2048" }
---
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: assistant
  namespace: ml-team
spec:
  endpoints:
  - name: smollm2
    priority: 0
    selector:
      matchLabels:
        modelplane.ai/deployment: smollm2
  - name: smollm
    priority: 1
    selector:
      matchLabels:
        modelplane.ai/deployment: smollm

Wait for the deployment to report Ready:

kubectl -n ml-team wait md/smollm2 --for=condition=Ready --timeout=10m

Run the request again to show the switch:

curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model": "ml-team/assistant", "messages": [{"role": "user", "content": "What is Kubernetes?"}]}' \
  | jq -r '.model, .choices[0].message.content'

Now the model field shows ml-team/smollm2. Your request stayed the same, and so did the name it calls. You changed the ModelService, and the gateway picked up the new model.

Fall back to the old model

The first model still sits in assistant at priority 1. If SmolLM2 loses its replicas, the gateway sends requests to the first model. Scale SmolLM2 to zero replicas:

kubectl -n ml-team scale md smollm2 --replicas=0

Run the request again:

curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model": "ml-team/assistant", "messages": [{"role": "user", "content": "What is Kubernetes?"}]}' \
  | jq -r '.model, .choices[0].message.content'

The gateway takes a few seconds to move the route off SmolLM2. If the model still shows ml-team/smollm2, wait a moment and run the request again.

The model field shows ml-team/smollm again. Requests kept reaching a model while SmolLM2 had no replicas. Scale SmolLM2 back up:

kubectl -n ml-team scale md smollm2 --replicas=1
kubectl -n ml-team wait md/smollm2 --for=condition=Ready --timeout=10m

Watch it take the requests again:

curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model": "ml-team/assistant", "messages": [{"role": "user", "content": "What is Kubernetes?"}]}' \
  | jq -r '.model, .choices[0].message.content'

To retire the first model, remove its endpoint from assistant and delete its ModelDeployment. Callers keep using ml-team/assistant.

Clean up

Delete both clusters:

curl -fsSL https://docs.modelplane.ai/main/examples/getting-started/local-demo/setup.sh | bash -s -- --clean

Next steps

Take the tour to run Modelplane on a cloud account, with GPU clusters across regions.