zurawiki/tiktoken-rs

[tiktoken-rs] - Library for tokenizing text with OpenAI models using tiktoken. [](https://github.com/zurawiki/tiktoken-rs/actions/workflows/ci.yml)