WordInfo subsetting

It is possible to ask Suachi to return only a subset of fields in the WordInfo. To do that, you can use the fields parameter of the sudachipy.Dictionary.tokenizer() method. The parameter accepts a set of strings, each one representing a field to be returned. By default, all fields are returned.

Allowed values:

  • surface: in-dictionary surface word form.

    sudachipy.Morpheme.surface() method returns the slice of the input text and is not affected by that flag.

  • pos or pos_id: part-of-speech tag.

  • normalized_form

  • dictionary_form

  • reading_form

  • word_structure

  • synonym_group_ids

  • user_data

  • split_a

  • split_b

Note

If you want only tokenization (e.g. use only sudachipy.Morpheme.surface(), passing empty set is allowed.

You need to load splits if you want to use sudachipy.Morpheme.split() method. If performing the tokenization with non-default mode, the required splits will be loaded automatically and can be omitted.:

dic.tokenizer(SplitMode.B, fields=set()) # implicitly becomes fields={'split_b'}

To access user-defined dictionary data when field subsetting is enabled, load user_data explicitly.:

tokenizer = dic.tokenizer(fields={"surface", "user_data"})
morpheme = tokenizer.tokenize("すだち")[0]
assert morpheme.user_data() == "徳島県産"

If fields is specified without user_data, sudachipy.Morpheme.user_data() follows the same partial-loading rule as other fields and may return an incorrect result.

Warning

Using a field not included in the passed subset will produce an incorrect result without any warning. Use tests to ensure that the needed fields are loaded when using this parameter.