inference-platform

Here are 9 public repositories matching this topic...

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

python machine-learning deep-learning model-serving multimodal mlops ml-engineering ai-inference llm generative-ai llmops llm-serving model-inference-service llm-inference inference-platform

Updated Sep 7, 2026
Python

InftyAI / llmaz

Star

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

kubernetes inference huggingface llm modelscope llamacpp vllm text-generation-inference ollama sglang inference-platform

Updated Aug 27, 2026
Go

Lattica-ai / lattica-studio

Star

A tool for developers and researchers to design FHE pipelines and execute them on cloud-based GPUs

cryptography inference post-quantum-cryptography homomorphic-encryption post-quantum fhe ai-security accelerated-computing fully-homomorphic-encryption confidential-computing post-quantum-security post-quantum-algorithms ai-tools secure-inference post-quantum-encryption inference-platform post-quantum-cryptography-

Updated Sep 23, 2026
Python

MrScripty / Pantograph

Star

Local AI workflow orchestration and runtime management framework

workflow-engine embeddings tts stt observability model-serving 3d-generation depth-mapping llm node-based-ui generative-ai audio-diffusion local-ai inference-platform image-diffusion

Updated Sep 9, 2026
Rust

lee-cheng-han / gpu-aware-llm-serving-platform

Star

GPU-aware LLM serving platform with multi-worker routing, dynamic batching, model placement, streaming, admission control, and reproducible performance benchmarks.

distributed-systems pytorch model-serving fastapi huggingface-transformers dynamic-batching gpu-scheduling llm-serving inference-platform ai-infrastructure

Updated Sep 9, 2026
Python

Sourajit-M / local-ai-inference-platform

Star

Local AI inference platform built with FastAPI, React, Ollama, and SQLite. Features multi-model benchmarking, structured generation, experiment tracking, JWT authentication, and performance analytics for small language models running entirely offline.

react machine-learning typescript jwt-authentication fastapi ai-engineering llm ollama offline-ai inference-platform

Updated Jun 16, 2026
TypeScript

gauravkukal / sensei-agentic-inference-orchestrator

Star

A conceptual framework for a high-scale Agentic AI orchestrator, inspired by enterprise-grade inference platforms.

distributed-systems ml-ops generative-ai inference-platform agentic-ai

Updated Mar 15, 2026

TheStableFoundation / stbl-protocol

Star

artificial-intelligence inference-platform web333

Updated Jan 7, 2026
Rust

CeciliaGit / ai-inference-platform-lab

Star

AI inference platform architecture lab demonstrating admission control, fairness scheduling, bounded queues, and graceful degradation under burst traffic.

distributed-systems microservices load-testing prometheus observability system-design platform-architecture inference-platform ai-infrastructure rag-architecture

Updated Mar 15, 2026
Python

Add this topic to your repo

To associate your repository with the inference-platform topic, visit your repo's landing page and select "manage topics."

Learn more