tokn 0.1.0
Given a script containing token descriptions (each a regular expression), tokn compiles an automaton which it can then use to efficiently convert a text file to a sequence of those tokens.
Given a script containing token descriptions (each a regular expression), tokn compiles an automaton which it can then use to efficiently convert a text file to a sequence of those tokens.