shimmy

github.com ↗

Python-free Rust inference server for NLP models with OpenAI API compatibility and hot model swapping.