Struct InputBuffer

pub struct InputBuffer { /* private fields */ }

InputBuffer - prepares the input data for the analysis

By saying char we actually mean Unicode codepoint here. In the context of this struct these terms are synonyms.

Implementations

impl InputBuffer

pub fn new() -> InputBuffer

Creates new InputBuffer

pub fn reset(&mut self) -> &mut String

Resets the input buffer, so it could be used to process new input. New input should be written to the returned mutable reference.

pub fn from<T: AsRef<str>>(data: T) -> InputBuffer

Creates input from the passed string. Should be used mostly for tests.

Panics if the input string is too long.

pub fn start_build(&mut self) -> SudachiResult<()>

Moves InputBuffer into RW state, making it possible to perform edits on it

pub fn build(&mut self, grammar: &Grammar<'_>) -> SudachiResult<()>

Finalizes InputBuffer state, making it RO

pub fn with_editor<'a, F>(&mut self, func: F) -> SudachiResult<()>

where F: FnOnce(&InputBuffer, InputEditor<'a>) -> SudachiResult<InputEditor<'a>> + 'a,

Execute a function which can modify the contents of the current buffer

Edit can borrow &str from the context with the borrow checker working correctly

pub fn refresh_chars(&mut self)

Recompute chars from modified string (useful if the processing will use chars)

impl InputBuffer

pub fn original(&self) -> &str

Borrow original data

pub fn current(&self) -> &str

Borrow modified data

pub fn current_chars(&self) -> &[char]

Borrow array of current characters

pub fn curr_byte_offsets(&self) -> &[usize]

Returns byte offsets of current chars

pub fn get_original_index(&self, index: usize) -> usize

Get index of the current byte in original sentence Bytes not on character boundaries are not supported

pub fn to_orig_byte_idx(&self, index: usize) -> usize

Mod Char Idx -> Orig Byte Idx

pub fn to_orig_char_idx(&self, index: usize) -> usize

Mod Char Idx -> Orig Char Idx

pub fn to_curr_byte_idx(&self, index: usize) -> usize

Mod Char Idx -> Mod Byte Idx

pub fn curr_slice_c(&self, data: Range<usize>) -> &str

Input: Mod Char Idx

pub fn orig_slice_c(&self, data: Range<usize>) -> &str

Input: Mod Char Idx

pub fn ch_idx(&self, idx: usize) -> usize

pub fn swap_original(&mut self, target: &mut String)

Swaps original data with the passed location

pub fn into_original(self) -> String

Return original data as owned, consuming itself

pub fn can_bow(&self, offset: usize) -> bool

Whether the byte can start a new word. Supports bytes not on character boundaries.

pub fn get_word_candidate_length(&self, char_idx: usize) -> usize

Returns char length to the next can_bow point

Used by SimpleOOV plugin

Trait Implementations

impl Clone for InputBuffer

fn clone(&self) -> InputBuffer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for InputBuffer

fn default() -> InputBuffer

Returns the “default value” for a type.

impl InputTextIndex for InputBuffer

fn cat_of_range(&self, range: Range<usize>) -> CategoryType

Common character category inside the range. Indexed by chars.

fn cat_at_char(&self, offset: usize) -> CategoryType

Character category at char offset

fn cat_continuous_len(&self, offset: usize) -> usize

Number of chars to the right of the offset with the same character category

fn char_distance(&self, cpt: usize, offset: usize) -> usize

Distance in chars between the char indexed by index and the char, relative to it by offset. Java name: getCodePointsOffsetLength

fn orig_slice(&self, range: Range<usize>) -> &str

Returns substring of original text by indices from the current text

fn curr_slice(&self, range: Range<usize>) -> &str

Returns substring of the current (modified) text by indices from the current text

fn to_orig(&self, range: Range<usize>) -> Range<usize>

Translate range from current state to original. Byte-indexed.