Module stateless_tokenizer 

Structs

StatelessTokenizer

Implementation of a Tokenizer which does not have tokenization state.

Traits

DictionaryAccess

Provides access to dictionary data