Back to List
Notice:This resource is provided by a third-party author. Please review the code with AI tools or manually before use to ensure security and compatibility.
Rusthuggingface/tokenizers

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

89.3/100
10.8KForks: 1.1K
View on GitHubHomepage →
Loading report...

Similar Projects

rust-bert

57

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

Rust3.1K

memvid

87

Memory layer for AI Agents. Replace complex RAG pipelines with a serverless, single-file memory layer. Give your agents instant retrieval and long-term memory.

Rust15.6K

tensorzero

88

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation.

Rust11.5K

code2prompt

88

A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.

Rust7.4K
Back to List