sudachipy.tokenizer package

Note

  • Import from sudachipy.tokenizer is deprecated.
    • Use from sudachipy import Tokenizer instead.

    • You can also import SplitMode: from sudachipy import SplitMode.

Module contents

class sudachipy.tokenizer.Tokenizer

Sudachi Tokenizer, Python version

SplitMode = SplitMode.C
mode
tokenize($self, text: str, mode = None, logger = None, out = None) sudachipy.MorphemeList

Break text into morphemes.

SudachiPy 0.5.* had logger parameter, it is accepted, but ignored.

Parameters: