Module stateless_tokenizer

Structs

• StatelessTokenizer
  Implementation of a Tokenizer which does not have tokenization state.

Traits

• DictionaryAccess
  Provides access to dictionary data