discourse_ai-tokenizers 0.1.0

A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.

Gemfile:
=

instalar:
=

Versões:

0.3.1 July 07, 2025 (20,6 MB)
0.3.0 July 04, 2025 (20,6 MB)
0.2.0 July 02, 2025 (20,6 MB)
0.1.2 July 02, 2025 (20,6 MB)
0.1.1 July 01, 2025 (20,6 MB)
0.1.0 July 01, 2025 (20,6 MB)

Mostrar todas as versões (6)

Runtime Dependencies (3):

activesupport >= 6.0

tiktoken_ruby ~> 0.0.11.1

tokenizers ~> 0.5.4

Development Dependencies (2):

rubocop-discourse = 3.8.1

syntax_tree ~> 6.2.0

Mostrar todas as dependências

Donos:

Pushed by:

Autores:

Rafael Silva

discourse_ai-tokenizers 0.1.0

Gemfile:
=

instalar:
=

Versões:

Runtime Dependencies (3):

Development Dependencies (2):

Donos:

Pushed by:

Autores:

SHA 256 checksum:

Total de downloads 3.004

Desta versão 128

Versão lançada:
July 1, 2025 4:57pm

Licença:
MIT

Versão Requerida do Ruby: >= 3.2.0

Links:

discourse_ai-tokenizers 0.1.0

Gemfile: =

instalar: =

Versões:

Runtime Dependencies (3):

Development Dependencies (2):

Donos:

Pushed by:

Autores:

SHA 256 checksum:

Total de downloads 3.004

Desta versão 128

Versão lançada: July 1, 2025 4:57pm

Licença: MIT

Versão Requerida do Ruby: >= 3.2.0

Links:

Gemfile:
=

instalar:
=

Versão lançada:
July 1, 2025 4:57pm

Licença:
MIT