FlashInfer: Kernel Library for LLM Serving
-
Updated
Sep 23, 2026 - Cuda
#
distributed-inference
Here are 146 public repositories matching this topic...
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
-
Updated
Sep 23, 2026 - Python
Achieve state of the art inference performance with modern accelerators on Kubernetes
-
Updated
Sep 23, 2026 - Shell
SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
text-to-speech streaming cuda inference pytorch tts speech-recognition music-generation asr model-serving multimodal openai-api audio-generation sglang distributed-inference
-
Updated
Sep 23, 2026 - Python
Best practices, reference architectures, and examples for distributed AI training and inference on AWS.
kubernetes aws gpu slurm efa distributed-training eks parallelcluster hyperpod generative-ai distributed-inference physical-ai
-
Updated
Sep 22, 2026 - Shell
Every device brings a slice. Together they run the whole model. Peer-to-peer LLM inference across browser tabs: a from-scratch WebGPU engine and a WebRTC runtime that split a 27B model over the devices in a room.
browser webrtc peer-to-peer inference webgpu wgsl llm local-llm speculative-decoding distributed-inference
-
Updated
Sep 20, 2026 - JavaScript
Split FLUX.2 and LTX 2.3 across two GPUs (LAN or same-machine) — NVENC compresses activations live on the wire. Icarus (ComfyUI node) + Daedalus (back-half server).
flux pytorch image-generation multi-gpu nvenc video-generation pipeline-parallelism diffusion-models flux2 comfyui distributed-inference comfyui-node ltx-video
-
Updated
May 17, 2026 - Python
[Official] Prima.cpp: Scale Your Local AI Beyond One Device.
-
Updated
Sep 6, 2026 - C++
一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。
-
Updated
Aug 23, 2026 - JavaScript
An educational distributed training and inference library for neural nets using local computing
-
Updated
Jun 10, 2026 - Python
Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.
rust ai mcp decentralized p2p inference self-hosted candle libp2p huggingface openai-api llm chatgpt-alternative local-llm gguf anthropic-api distributed-inference claude-code ollama-alternative openclaw
-
Updated
Sep 23, 2026 - Rust
Mixed-vendor GPU inference cluster manager with speculative decoding
python machine-learning deep-learning metal cuda p2p homelab rocm gpu-cluster llama-cpp gpu-inference local-llm llm-inference ollama speculative-decoding distributed-inference gpu-pooling self-hosted-ai openai-compatible mixed-gpu
-
Updated
Jul 2, 2026 - Python
Run any model on Intel silicon
-
Updated
Sep 22, 2026 - Rust
MiniMax-H3 multi-GPU parallel inference for ComfyUI | 多卡并行加速节点:2-8 GPU Ulysses sequence parallel, bit-identical video+audio generation
h3 ulysses minimax multi-gpu multigpu video-generation image-to-video video-audio text-to-video multi-gpu-inference comfyui sequence-parallelism distributed-inference comfyui-custom-node minimax-h3
-
Updated
Aug 14, 2026 - Python
Turn any Mac or GPU into an OpenAI-compatible inference node. One-command setup, automatic HTTPS, model management, and distributed request routing.
machine-learning gpu inference nvidia cloudflare llama mistral apple-silicon llama-cpp local-llm qwen gguf deepseek distributed-inference self-hosted-ai openai-compatible
-
Updated
Sep 5, 2026 - Python
🧠 One LLM, split across a Mac (Apple MPS) + a Windows PC (NVIDIA CUDA) — heterogeneous pipeline inference over a framework-neutral wire, bit-for-bit identical to single-machine. No torch.distributed, no datacenter.
metal peer-to-peer decentralized cuda transformers inference pytorch mps gemma model-parallelism heterogeneous-computing edge-ai pipeline-parallelism on-device-ai apple-silicon large-language-models llm llm-inference qwen distributed-inference
-
Updated
Jul 6, 2026 - Python
Code for paper "JMDC: A Joint Model and Data Compression System for Deep Neural Networks Collaborative Computing in Edge-Cloud Networks"
compression deep-learning distributed-computing inference dnn neural-networks pruning quantization dnn-optimization edge-computing mobilenet edge-cloud ai-models dnn-partition distributed-inference
-
Updated
Jul 22, 2026 - Python
Optimize and run consistent AI Functions.
s3 prompt data-engineering data-pipelines observability evaluation-framework synthetic-data unstructured-data mlops batch-inference prompt-engineering llm-inference evals distributed-inference evals-engineering
-
Updated
Sep 22, 2026 - Python
Reproducible LLM inference on four NVIDIA DGX Spark (GB10) nodes wired as a direct switchless ConnectX ring — serving profiles, ring launchers, pinned ARM64 runtime images and measured results, newest tested model first.
arm64 llm-serving vllm sglang distributed-inference gb10 dgx-spark tensor-parallel switchless-ring nccl-mesh
-
Updated
Sep 14, 2026 - Python
Add this topic to your repo
To associate your repository with the distributed-inference topic, visit your repo's landing page and select "manage topics."