pub struct StatelessTokenizer<T> { /* private fields */ }Expand description
Implementation of a Tokenizer which does not have tokenization state.
This is a wrapper which is generic over dictionary pointers. Usable where dictionary is a struct itself, &, &mut, Rc<.>, Arc<.>.
Implementations§
source§impl<T: DictionaryAccess> StatelessTokenizer<T>
impl<T: DictionaryAccess> StatelessTokenizer<T>
pub fn new(dict: T) -> StatelessTokenizer<T>
Trait Implementations§
source§impl<T> Tokenize for StatelessTokenizer<T>where
T: DictionaryAccess + Clone,
impl<T> Tokenize for StatelessTokenizer<T>where
T: DictionaryAccess + Clone,
type Dictionary = T
source§fn tokenize<'a>(
&'a self,
input: &'a str,
mode: Mode,
enable_debug: bool,
) -> SudachiResult<MorphemeList<Self::Dictionary>>
fn tokenize<'a>( &'a self, input: &'a str, mode: Mode, enable_debug: bool, ) -> SudachiResult<MorphemeList<Self::Dictionary>>
Break text into
MorphemesAuto Trait Implementations§
impl<T> Freeze for StatelessTokenizer<T>where
T: Freeze,
impl<T> RefUnwindSafe for StatelessTokenizer<T>where
T: RefUnwindSafe,
impl<T> Send for StatelessTokenizer<T>where
T: Send,
impl<T> Sync for StatelessTokenizer<T>where
T: Sync,
impl<T> Unpin for StatelessTokenizer<T>where
T: Unpin,
impl<T> UnwindSafe for StatelessTokenizer<T>where
T: UnwindSafe,
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
source§impl<T> IntoEither for T
impl<T> IntoEither for T
source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moresource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more