token_estimator 0.1.0
TokenEstimator is a Rails gem that allows you to count tokens in Excel, CSV, PDF, TXT, Markdown, and input text files using different tokenizers.
Gemfile:
=
install:
=
Runtime Dependencies (7):
csv
~> 3.3
kramdown
~> 2.4
nokogiri
~> 1.16, >= 1.16.6
pdf-reader
~> 2.12
rails
>= 7.2.0.beta2
roo
~> 2.10, >= 2.10.1
tokenizers
~> 0.5.0