LLMKube：管理混合GPU上的本地大模型

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Run…

LLMKube The Kubernetes operator for self-hosted LLM inference Run models across NVIDIA, Apple Silicon, and AMD from one spec. Your models. Your hardware. Your rules. Three things no other Kubernetes LLM operator combines: Apple Silicon inference as native macOS processes, not containers (the Metal Agent runs llama-server, mlx-server, and vllm-swift with full Metal GPU access); heterogeneous fleets of NVIDIA, AMD (Vulkan), Intel, and Apple Silicon hosts under one operator and one set of CRDs; and fail-closed policy routing between local and external models via ModelRouter. Who it's for: platform and AI-ops teams running on-prem, mixed-hardware, or air-gapped inference. If you want one model on one laptop, Ollama is simpler. Plus Foreman: an agentic harness that runs coding agents on your own fleet. It has shipped real fixes to this repo. Quick Start • Status • ModelRouter • Foreman • Metal Agent • Why LLMKube? • Benchmarks • Roadmap • Discord The Problem You want to run LLMs on your own infrastructure. Maybe it's for data privacy, cost control, air-gapped compliance, or you just don't want to send every request to OpenAI. So you set up llama.cpp. It works great on one machine. Then you need to scale it, monitor it, manage model versions, handle GPU scheduling across nodes, expose an API, and somehow make your Mac's Metal GPU and your Linux server's NVIDIA cards work together. And the moment you want any of that traffic to sometimes hand off to Claude or GPT, you're building another routing layer. Suddenly you're building an entire platform instead of shipping your product. LLMKube is a Kubernetes operator that turns LLM deployment into a two-line YAML problem. Define a Model and an InferenceService, and the operator handles downloading, caching, GPU scheduling, health checks, scaling, and exposing an OpenAI-compatible API. Add a ModelRouter on top and the same cluster does policy-aware routing between your local models and external providers (Anthropic / OpenAI / LiteLLM) with fail-closed semantics for regulated data. Architecture Two cooperating processes. An in-cluster controller owns Kubernetes-side desired state. An out-of-cluster metal-agent (optional, only needed for Apple Silicon hosts) owns OS-level process supervision and registers its authenticated ingress back into the cluster; the controller fronts that ingress with an in-cluster relay, so Metal InferenceServices are ordinary Services. %%{init: {'theme':'neutral','flowchart':{'curve':'linear'}}}%% flowchart TB subgraph CLUSTER["Kubernetes cluster"] direction LR CTRL["LLMKube controller"] CRD["Model · InferenceService<br/>(custom resources)"] POD["Runtime pods<br/>llama.cpp · vLLM · TGI"] RELAY["Relay pod<br/>(per Metal InferenceService)"] CRD -- watched by --> CTRL CTRL -- schedules --> POD CTRL -- creates --> RELAY end subgraph HOST["Apple Silicon host (optional)"] direction LR AGENT["metal-agent<br/>(TLS ingress)"] NATIVE["llama-server · mlx-server · vllm-swift<br/>(native processes)"] AGENT -- supervises --> NATIVE end AGENT -- "registers ingress" --> CLUSTER RELAY -- "pinned TLS + token" --> AGENT Loading Same operator manages Linux/GPU pods and Apple Silicon hosts; both surface as InferenceService objects to kubectl. Setup guide for the metal-agent on Apple Silicon: deployment/macos/README.md. See it in action Live asciinema cast on llmkube.com/docs/getting-started: deploy a model on a kind cluster, stream tokens from the OpenAI-compatible endpoint, and run the built-in throughput benchmark in under a minute. Quick Start # Install the CLI brew install defilantech/tap/llmkube # Install the operator on any K8s cluster helm repo add llmkube https://defilantech.github.io/LLMKube helm install llmkube llmkube/llmkube --namespace llmkube-system --create-namespace # Deploy a model (one command, uses catalog-tested defaults) llmkube deploy phi-4-mini # Query it (OpenAI-compatible) kubectl port-forward svc/phi-4-mini 8080:8080 & curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":100}' That's it. The operator downloads the model, creates the deployment, sets up the service, and exposes an OpenAI-compatible API. Works with the OpenAI Python/Node/Go SDKs, LangChain, and LlamaIndex out of the box. Want GPU acceleration? Add --gpu: llmkube deploy llama-3.1-8b --gpu --gpu-count 1 No CLI? Use plain kubectl apiVersion: inference.llmkube.dev/v1alpha1 kind: Model metadata: name: tinyllama spec: source: https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf format: gguf --- apiVersion: inference.llmkube.dev/v1alpha1 kind: InferenceService metadata: name: tinyllama spec: modelRef: tinyllama replicas: 1 resources: cpu: "500m" memory: "1Gi" kubectl apply -f model.yaml Full setup guides: Minikube Quickstart | GKE with GPUs | Intel GPU Quickstart | DGX Spark (MicroK8s) | Air-Gapped Deployment | OpenShift Project status Actively maintained: frequent point releases, three in October 2026 alone as of 2026-10-16 (v0.10.2 through v0.10.4). See the latest release and the CHANGELOG for what's current. First-time contributors have landed real features: Intel oneAPI / SYCL GPU support arrived as PR #557 from a first-time contributor, and configurable GPU resource names for AMD/Vulkan/Intel scheduling as #709 from a contributor new to the repo, who followed with vendor-neutral DRA scheduling in #750. Foreman, the agentic harness, contributes code to this repository: its first two merged PRs landed in May 2026 (#508, #588), with more Foreman-authored PRs merging since. Running LLMKube in production, a lab, or a homelab? Add yourself to ADOPTERS.md. The Metal Agent This is the thing no other Kubernetes LLM tool does. Most Kubernetes tools run inference inside containers. That works fine on Linux with NVIDIA GPUs. But Apple Silicon's Metal GPU can't be accessed from inside a container — so every other to