Struct WordSegmenter

pub struct WordSegmenter { /* private fields */ }

Supports loading word break data, and creating word break iterators for different string encodings.

Most segmentation methods live on WordSegmenterBorrowed, which can be obtained via WordSegmenter::new_auto() (etc) or WordSegmenter::as_borrowed().

Examples

Segment a string:

use icu::segmenter::{options::WordBreakInvariantOptions, WordSegmenter};
let segmenter =
    WordSegmenter::new_auto(WordBreakInvariantOptions::default());

let breakpoints: Vec<usize> =
    segmenter.segment_str("Hello World").collect();
assert_eq!(&breakpoints, &[0, 5, 6, 11]);

Segment a Latin1 byte string:

use icu::segmenter::{options::WordBreakInvariantOptions, WordSegmenter};
let segmenter =
    WordSegmenter::new_auto(WordBreakInvariantOptions::default());

let breakpoints: Vec<usize> =
    segmenter.segment_latin1(b"Hello World").collect();
assert_eq!(&breakpoints, &[0, 5, 6, 11]);

Successive boundaries can be used to retrieve the segments. In particular, the first boundary is always 0, and the last one is the length of the segmented text in code units.

use itertools::Itertools;
let text = "Mark’d ye his words?";
let segments: Vec<&str> = segmenter
    .segment_str(text)
    .tuple_windows()
    .map(|(i, j)| &text[i..j])
    .collect();
assert_eq!(
    &segments,
    &["Mark’d", " ", "ye", " ", "his", " ", "words", "?"]
);

Not all segments delimited by word boundaries are words; some are interword segments such as spaces and punctuation. The WordBreakIterator::word_type() of a boundary can be used to classify the preceding segment; WordBreakIterator::iter_with_word_type() associates each boundary with its status.

let words: Vec<&str> = segmenter
    .segment_str(text)
    .iter_with_word_type()
    .tuple_windows()
    .filter(|(_, (_, segment_type))| segment_type.is_word_like())
    .map(|((i, _), (j, _))| &text[i..j])
    .collect();
assert_eq!(&words, &["Mark’d", "ye", "his", "words"]);

Implementations

impl WordSegmenter

pub fn new_auto( _options: WordBreakInvariantOptions, ) -> WordSegmenterBorrowed<'static>

Constructs a WordSegmenter with an invariant locale and the best available compiled data for complex scripts (Chinese, Japanese, Khmer, Lao, Myanmar, and Thai).

The current behavior, which is subject to change, is to use the LSTM model when available and the dictionary model for Chinese and Japanese.

✨ Enabled with the compiled_data and auto Cargo features.

📚 Help choosing a constructor

Examples

Behavior with complex scripts:

use icu::segmenter::{options::WordBreakInvariantOptions, WordSegmenter};

let th_str = "ทุกสองสัปดาห์";
let ja_str = "こんにちは世界";

let segmenter =
    WordSegmenter::new_auto(WordBreakInvariantOptions::default());

let th_bps = segmenter.segment_str(th_str).collect::<Vec<_>>();
let ja_bps = segmenter.segment_str(ja_str).collect::<Vec<_>>();

assert_eq!(th_bps, [0, 9, 18, 39]);
assert_eq!(ja_bps, [0, 15, 21]);

pub fn try_new_auto(options: WordBreakOptions<'_>) -> Result<Self, DataError>

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

pub fn try_new_auto_with_buffer_provider( provider: &(impl BufferProvider + ?Sized), options: WordBreakOptions<'_>, ) -> Result<Self, DataError>

A version of [Self :: try_new_auto] that uses custom data provided by a BufferProvider.

✨ Enabled with the serde feature.

📚 Help choosing a constructor

pub fn try_new_auto_unstable<D>( provider: &D, options: WordBreakOptions<'_>, ) -> Result<Self, DataError>

where D: DataProvider<SegmenterBreakWordV1> + DataProvider<SegmenterBreakWordOverrideV1> + DataProvider<SegmenterDictionaryAutoV1> + DataProvider<SegmenterLstmAutoV1> + DataProvider<SegmenterBreakGraphemeClusterV1> + ?Sized,

A version of Self::new_auto that uses custom data provided by a DataProvider.

📚 Help choosing a constructor

⚠️ The bounds on provider may change over time, including in SemVer minor releases.

pub fn new_lstm( _options: WordBreakInvariantOptions, ) -> WordSegmenterBorrowed<'static>

Constructs a WordSegmenter with an invariant locale and compiled LSTM data for complex scripts (Burmese, Khmer, Lao, and Thai).

The LSTM, or Long Term Short Memory, is a machine learning model. It is smaller than the full dictionary but more expensive during segmentation (inference).

Warning: there is not currently an LSTM model for Chinese or Japanese, so the WordSegmenter created by this function will have unexpected behavior in spans of those scripts.

✨ Enabled with the compiled_data and lstm Cargo features.

📚 Help choosing a constructor

Examples

Behavior with complex scripts:

use icu::segmenter::{options::WordBreakInvariantOptions, WordSegmenter};

let th_str = "ทุกสองสัปดาห์";
let ja_str = "こんにちは世界";

let segmenter =
    WordSegmenter::new_lstm(WordBreakInvariantOptions::default());

let th_bps = segmenter.segment_str(th_str).collect::<Vec<_>>();
let ja_bps = segmenter.segment_str(ja_str).collect::<Vec<_>>();

assert_eq!(th_bps, [0, 9, 18, 39]);

// Note: We aren't able to find a suitable breakpoint in Chinese/Japanese.
assert_eq!(ja_bps, [0, 21]);

pub fn try_new_lstm(options: WordBreakOptions<'_>) -> Result<Self, DataError>

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

pub fn try_new_lstm_with_buffer_provider( provider: &(impl BufferProvider + ?Sized), options: WordBreakOptions<'_>, ) -> Result<Self, DataError>

A version of [Self :: try_new_lstm] that uses custom data provided by a BufferProvider.

✨ Enabled with the serde feature.

📚 Help choosing a constructor

pub fn try_new_lstm_unstable<D>( provider: &D, options: WordBreakOptions<'_>, ) -> Result<Self, DataError>

where D: DataProvider<SegmenterBreakWordV1> + DataProvider<SegmenterBreakWordOverrideV1> + DataProvider<SegmenterLstmAutoV1> + DataProvider<SegmenterBreakGraphemeClusterV1> + ?Sized,

A version of Self::new_lstm that uses custom data provided by a DataProvider.

📚 Help choosing a constructor

⚠️ The bounds on provider may change over time, including in SemVer minor releases.

pub fn new_dictionary( _options: WordBreakInvariantOptions, ) -> WordSegmenterBorrowed<'static>

Construct a WordSegmenter with an invariant locale and compiled dictionary data for complex scripts (Chinese, Japanese, Khmer, Lao, Myanmar, and Thai).

The dictionary model uses a list of words to determine appropriate breakpoints. It is faster than the LSTM model but requires more data.

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

Examples

Behavior with complex scripts:

use icu::segmenter::{options::WordBreakInvariantOptions, WordSegmenter};

let th_str = "ทุกสองสัปดาห์";
let ja_str = "こんにちは世界";

let segmenter =
    WordSegmenter::new_dictionary(WordBreakInvariantOptions::default());

let th_bps = segmenter.segment_str(th_str).collect::<Vec<_>>();
let ja_bps = segmenter.segment_str(ja_str).collect::<Vec<_>>();

assert_eq!(th_bps, [0, 9, 18, 39]);
assert_eq!(ja_bps, [0, 15, 21]);

pub fn try_new_dictionary( options: WordBreakOptions<'_>, ) -> Result<Self, DataError>

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

pub fn try_new_dictionary_with_buffer_provider( provider: &(impl BufferProvider + ?Sized), options: WordBreakOptions<'_>, ) -> Result<Self, DataError>

A version of [Self :: try_new_dictionary] that uses custom data provided by a BufferProvider.

✨ Enabled with the serde feature.

📚 Help choosing a constructor

pub fn try_new_dictionary_unstable<D>( provider: &D, options: WordBreakOptions<'_>, ) -> Result<Self, DataError>

where D: DataProvider<SegmenterBreakWordV1> + DataProvider<SegmenterBreakWordOverrideV1> + DataProvider<SegmenterDictionaryAutoV1> + DataProvider<SegmenterDictionaryExtendedV1> + DataProvider<SegmenterBreakGraphemeClusterV1> + ?Sized,

A version of Self::new_dictionary that uses custom data provided by a DataProvider.

📚 Help choosing a constructor

⚠️ The bounds on provider may change over time, including in SemVer minor releases.

pub fn as_borrowed(&self) -> WordSegmenterBorrowed<'_>

Constructs a borrowed version of this type for more efficient querying.

Most useful methods for segmentation are on this type.

Trait Implementations

impl Debug for WordSegmenter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.