sudachi::analysis

Module stateless_tokenizer

Source

Structs§

StatelessTokenizer
Implementation of a Tokenizer which does not have tokenization state.

Traits§

DictionaryAccess
Provides access to dictionary data