Struct TextNormalizer 

pub struct TextNormalizer<D = DefaultInputTextPlugin> { /* private fields */ }

Applies input-text normalization used by tokenizer input processing.

By default, this uses DefaultInputTextPlugin. When built from a dictionary, it applies that dictionary’s configured input-text plugins.

Implementations

impl TextNormalizer<DefaultInputTextPlugin>

pub fn new(grammar: &Grammar<'_>) -> SudachiResult<Self>

Create a text normalizer using the default input-text plugin.

pub fn try_default() -> SudachiResult<Self>

Create a text normalizer using the default input-text plugin and an empty grammar.

pub fn normalize(&mut self, text: &str) -> SudachiResult<String>

impl<D> TextNormalizer<D>

where D: DictionaryAccess,

pub fn from_dictionary(dictionary: D) -> Self

Create a text normalizer using the input-text plugins from a dictionary.

pub fn normalize(&mut self, text: &str) -> SudachiResult<String>

impl<D> TextNormalizer<Arc<D>>

where D: DictionaryAccess + ?Sized,

pub fn from_shared_dictionary(dictionary: Arc<D>) -> Self

Create a text normalizer using the input-text plugins from a shared dictionary handle.