Inference & Serving

SparkEngineAI/

QuantClaw-plugin

QuantClaw is a plug-and-play task-type routing quantization plugin for OpenClaw.

RightNow-AI/

AutoMegaKernel

An agent harness that compiles a model into one provably-correct, self-retargeting CUDA megakernel and self-tunes it past cuBLAS at batch-1 LLM decode, paper: https://arxiv.org/abs/2606.09682

generative-computing/

granite-switch

Granite Switch — Build AI models like you build software

PMZFX/

intel-arc-pro-b70-benchmarks

Benchmark results and performance data for the Intel Arc Pro B70 GPU (Xe2/Battlemage) - LLM inference, video generation, dual-GPU scaling.

OpenCPIL/

prima.cpp

[Official] prima.cpp: Fast 30-70B LLM inference on heterogeneous and everyday home devices

MADEVAL/

FerryAI

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embeddings, RAG and vector search. No Python, no HTTP microservices, no Docker sidecars.

wuxuedaifu/

vllm-chatterbox-stream

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages.

AdamBien/

lightmetal

Apple Silicon mlx with Zero Dependency Java

giannisanni/

pulsar

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.