sudachipy package

config.Config

class sudachipy.config.Config(system: str = None, user: list[str] = None, projection: str = 'surface', connectionCostPlugin: list = None, oovProviderPlugin: list = None, pathRewritePlugin: list = None, inputTextPlugin: list = None, characterDefinitionFile: str = None)[source]

SudachiPy rich configuration object.

Fields passed here will override the fields in the default configuration.

as_jsons()[source]

Convert this Config object to the json string

projection: str = 'surface'

Output the following field as the result of sudachipy.Morpheme.surface() instead of its value. This option works for pre-tokenizers created for a given dictionary as well. The original value is available as sudachipy.Morpheme.raw_surface().

This option is created for chiTra integration.

Available options:

  • surface

  • normalized

  • reading

  • dictionary

  • dictionary_and_surface

  • normalized_and_surface

  • normalized_nouns

system: str = None

Path to dictionary or one of three strings: ‘small’, ‘core’, ‘full’. If the file with the specified path does not exist and is not one of three special values, raise an error. If you want to use dictionary with one of predefined names, use relative paths e.g. ‘./core’ instead of ‘core’.

If the value is one of three special values and there does not exist a file with the same name, we try to load the dictionary from the SudachiDict_{system} installed package. For example, for “core” we will try to load the dictionary from the installed SudachiDict_core package.

user: list[str] = None

Paths to user dictionaries, maximum number of user dictionaries is 14

Dictionary

  • Dictionary does not provide an access to the grammar and lexicon.

class sudachipy.Dictionary(config_path=None, resource_dir=None, dict=None, dict_type=None, *, config=None) → Dictionary

–

A sudachi dictionary.

If both config.systemDict and dict are not given, sudachidict_core is used. If both config.systemDict and dict are given, dict is used. If dict is an absolute path to a file, it is used as a dictionary.

Parameters:

  • config_path (Config | pathlib.Path | str | None) – path to the configuration JSON file, config json as a string, or a [sudachipy.Config] object.

  • config (Config | pathlib.Path | str | None) – alias to config_path, only one of them can be specified at the same time.

  • resource_dir (pathlib.Path | str | None) – path to the resource directory folder.

  • dict (pathlib.Path | str | None) – type of pre-packaged dictionary, referring to sudachidict_<dict> packages on PyPI: https://pypi.org/search/?q=sudachidict. Also, can be an _absolute_ path to a compiled dictionary file.

  • dict_type (pathlib.Path | str | None) – deprecated alias to dict.

close()

Close this dictionary.

create(self, /, mode=SplitMode.C, fields=None, *, projection=None) → Tokenizer

–

Creates a sudachi tokenizer.

This method is deprecated, use Dictionary.tokenizer() instead.

Parameters:
entries(self, /) → Iterator[Morpheme]

–

Iterates over dictionary entries as standalone morphemes.

This iterates public lexicon entries. It includes entries that are referred to from other entries, such as split or constituent units, even when they are not indexed for normal lookup. Internal entries automatically generated for literal normalized forms are not exposed. The iteration order is not part of the public contract.

lookup(self, /, surface, out=None) → MorphemeList

–

Look up morphemes in the binary dictionary without performing the analysis.

The given surface is normalized before lookup. All morphemes from the dictionary with the normalized surface string are returned, with the last user dictionary searched first and the system dictionary searched last. Inside a dictionary, morphemes are outputted in-binary-dictionary order. Morphemes which are not indexed are not returned.

Parameters:
lookup_all_entries(self, /, surface, out=None) → MorphemeList

–

Look up morphemes by scanning all dictionary entries.

The given surface is normalized before matching. This scans public lexicon entries and can find entries which are not indexed for normal lookup. This can be slow on large dictionaries; use lookup() for normal indexed lookup.

Parameters:

  • surface (str) – find all morphemes whose normalized surface matches this value

  • out (MorphemeList | None) – if passed, reuse the given morpheme list instead of creating a new one.

oov_morpheme(self, /, pos_id, surface, reading=None, normalized_form=None, dictionary_form=None) → Morpheme

–

Create an out-of-vocabulary morpheme from the POS id and string forms.

Begin/end are set from the surface. When optional string forms are not provided, the surface is used for them.

Parameters:

  • pos_id (int) – part-of-speech id of the morpheme

  • surface (str) – surface of the morpheme

  • reading (str | None) – reading form of the morpheme

  • normalized_form (str | None) – normalized form of the morpheme

  • dictionary_form (str | None) – dictionary form of the morpheme

pos_matcher(target)

Creates a POS matcher object

If target is a function, then it must return whether a POS should match or not. If target is a list, it should contain partially specified POS. By partially specified it means that it is possible to omit POS fields or use None as a sentinel value that matches any POS.

For example, (‘名詞’,) will match any noun and (None, None, None, None, None, ‘終止形‐一般’) will match any word in 終止形‐一般 conjugation form.

Parameters:

target (Iterable[PartialPOS] | Callable[[POS], bool]) – can be either a list of POS partial tuples or a callable which maps POS to bool.

pos_of(self, /, pos_id: int) → tuple[str, str, str, str, str, str] | None

–

Returns POS with the given id.

Parameters:

pos_id (int) – POS id

Returns:

POS tuple with the given id or None for non existing id.

pre_tokenizer(self, /, mode=None, fields=None, handler=None, *, projection=None) → tokenizers.PreTokenizer

–

Creates HuggingFace Tokenizers-compatible PreTokenizer. Requires package tokenizers to be installed.

Parameters:
text_normalizer(self, /) → TextNormalizer

–

Creates a text normalizer from this dictionary.

The returned normalizer applies the same input-text plugins that this dictionary uses before tokenization.

tokenizer(self, /, mode=SplitMode.C, fields=None, *, projection=None) → Tokenizer

–

Creates a sudachi tokenizer.

Parameters:

TextNormalizer

TextNormalizer applies input-text plugins to raw input text. It does not perform morphological analysis or return morpheme normalized forms. Without a dictionary, it uses the default input-text normalization.

class sudachipy.TextNormalizer(dictionary=None) → TextNormalizer

–

A text normalizer.

This applies input-text plugins to raw input text. It does not perform morphological analysis or return morpheme normalized forms.

Create using Dictionary.text_normalizer() or by passing a Dictionary to this class. Without a dictionary, this uses the default input-text normalization.

normalize(self, /, text: str) → str

–

Normalize text using input-text plugins.

This normalizes tokenizer input text, not the dictionary-normalized form returned by Morpheme.normalized_form().

Parameters:

text (str) – text to normalize.

SplitMode

class sudachipy.SplitMode(mode=None) → SplitMode

–

Unit to split text.

A == short mode

B == middle mode

C == long mode

Parameters:

mode (str | None) – string representation of the split mode. One of [A,B,C] in captital or lower case. If None, returns SplitMode.C.

Tokenizer

class sudachipy.Tokenizer

A sudachi tokenizer

Create using Dictionary.create method.

SplitMode = SplitMode.C
mode

SplitMode of the tokenizer.

tokenize(self, /, text: str, mode=None, logger=None, out=None) → MorphemeList

–

Break text into morphemes.

Parameters:

  • text – text to analyze.

  • mode – analysis mode. This parameter is deprecated. Pass the analysis mode at the Tokenizer creation time and create different tokenizers for different modes. If you need multi-level splitting, prefer using Morpheme.split() method instead.

  • logger – Arg for v0.5.* compatibility. Ignored.

  • out – tokenization results will be written into this MorphemeList, a new one will be created instead. See https://worksapplications.github.io/sudachi.rs/python/topics/out_param.html for details.

A Tokenizer instance cannot be used concurrently from multiple threads. For parallel tokenization, share a Dictionary and create one Tokenizer per worker/thread. Concurrent calls raise sudachipy.errors.SudachiError.

Morpheme

  • Class method MorphemeList.empty() -> MorphemeList is deprecated.

    • Use Tokenizer.tokenize("") if you need.

class sudachipy.MorphemeList

A list of morphemes.

An object can not be instantiated manually. Use Tokenizer.tokenize(“”) to create an empty morpheme list.

classmethod empty(dict: Dictionary) → MorphemeList

–

Returns an empty morpheme list with dictionary.

Deprecated since version 0.6.0: Use Tokenizer.tokenize(“”) if you need.

get_internal_cost(self, /) → int

–

Returns the total cost of the path.

size(self, /) → int

–

Returns the number of morpheme in this list.

class sudachipy.Morpheme

A morpheme (basic semantic unit of language).

begin(self, /) → int

–

Returns the begin index of this in the input text.

dictionary_form(self, /) → str

–

Returns the dictionary form.

dictionary_form_morpheme(self, /) → Morpheme

–

Returns the morpheme corresponding to this morpheme’s dictionary form.

For out-of-vocabulary morphemes, invalid references, and references to the same dictionary entry, returns a morpheme equivalent to self. For a distinct referenced entry, returns a standalone morpheme whose begin/end offsets are 0..len(surface).

dictionary_id(self, /) → int

–

Returns the dictionary id which this word belongs.

end(self, /) → int

–

Returns the end index of this in the input text.

is_oov(self, /) → bool

–

Returns whether if this is out of vocabulary word.

normalized_form(self, /) → str

–

Returns the normalized form.

normalized_form_morpheme(self, /) → Morpheme

–

Returns the morpheme corresponding to this morpheme’s normalized form.

For out-of-vocabulary morphemes, invalid references, and references to the same dictionary entry, returns a morpheme equivalent to self. For a distinct referenced entry, returns a standalone morpheme whose begin/end offsets are 0..len(surface).

part_of_speech(self, /) → tuple[str, str, str, str, str, str]

–

Returns the part of speech as a six-element tuple. Tuple elements are four POS levels, conjugation type and conjugation form.

part_of_speech_id(self, /) → int

–

Returns the id of the part of speech in the dictionary.

raw_surface(self, /) → str

–

Returns the substring of input text corresponding to the morpheme regardless the configured projection.

See Config.projection.

reading_form(self, /) → str

–

Returns the reading form.

split(self, /, mode, out=None, add_single=False) → MorphemeList

–

Returns sub-morphemes in the provided split mode.

Parameters:

  • mode (SplitMode | None) – mode of new split.

  • out (MorphemeList | None) – write results to this MorphemeList instead of creating new one. See https://worksapplications.github.io/sudachi.rs/python/topics/out_param.html for more information on output parameters. Returned MorphemeList will be invalidated if this MorphemeList is used as an output parameter.

  • add_single (bool) – return lists with the current morpheme if the split hasn’t produced any elements. When False is passed, empty lists are returned instead.

surface(self, /) → str

–

Returns the substring of input text corresponding to the morpheme, or a projection if one is configured.

See Config.projection.

synonym_group_ids(self, /) → List[int]

–

Returns the list of synonym group ids.

user_data(self, /) → str

–

Returns user-defined data associated with this morpheme.

word_id(self, /) → int

–

Returns word id of this word in the dictionary.

WordInfo