A PHP implementation of OpenAI's BPE tokenizer tiktoken.
-
Updated
Jan 25, 2025 - PHP
#
special-tokens
Here are 3 public repositories matching this topic...
Detect and neutralize LLM chat-template special tokens (ChatML, Llama, Mistral, Gemma, Qwen, DeepSeek, Command-R) in untrusted text — zero-dependency CLI + library and CI gate against Special Token Injection.
cli zero-dependency ai-safety guardrails chat-template llm prompt-injection llm-security chatml special-tokens
-
Updated
Jun 21, 2026 - JavaScript
Custom BPE Tokenizer, trained for 1256 merges. Can be trained further, we can load the previously trained tokenizer and also there's an option to add or remove special tokens that you need.
-
Updated
Jun 23, 2025 - Python
Improve this page
Add a description, image, and links to the special-tokens topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the special-tokens topic, visit your repo's landing page and select "manage topics."