A pure Ruby implementation of OpenAI's tiktoken library for BPE tokenization

Required Ruby Version

>= 3.1.0

Authors

moznion

Versions

  1. 0.0.1 June 04, 2025 (14.5 KB)

Pushed by

moznion

SHA 256 checksum