Struct GraphemeClusterSegmenter

pub struct GraphemeClusterSegmenter { /* private fields */ }

Segments a string into grapheme clusters.

Supports loading grapheme cluster break data, and creating grapheme cluster break iterators for different string encodings.

Most segmentation methods live on GraphemeClusterSegmenterBorrowed, which can be obtained via GraphemeClusterSegmenter::new() or GraphemeClusterSegmenter::as_borrowed().

Examples

Segment a string:

use icu::segmenter::GraphemeClusterSegmenter;
let segmenter = GraphemeClusterSegmenter::new();

let breakpoints: Vec<usize> = segmenter.segment_str("Hello 🗺").collect();
// World Map (U+1F5FA) is encoded in four bytes in UTF-8.
assert_eq!(&breakpoints, &[0, 1, 2, 3, 4, 5, 6, 10]);

Segment a Latin1 byte string:

use icu::segmenter::GraphemeClusterSegmenter;
let segmenter = GraphemeClusterSegmenter::new();

let breakpoints: Vec<usize> =
    segmenter.segment_latin1(b"Hello World").collect();
assert_eq!(&breakpoints, &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]);

Successive boundaries can be used to retrieve the grapheme clusters. In particular, the first boundary is always 0, and the last one is the length of the segmented text in code units.

use itertools::Itertools;
let text = "मांजर";
let grapheme_clusters: Vec<&str> = segmenter
    .segment_str(text)
    .tuple_windows()
    .map(|(i, j)| &text[i..j])
    .collect();
assert_eq!(&grapheme_clusters, &["मां", "ज", "र"]);

This segmenter applies all rules provided to the constructor. Thus, if the data supplied by the provider comprises all grapheme cluster boundary rules from Unicode Standard Annex #29, Unicode Text Segmentation, which is the case of default data (both test data and data produced by icu_provider_source), the segment_* functions return extended grapheme cluster boundaries, as opposed to legacy grapheme cluster boundaries. See Section 3, Grapheme Cluster Boundaries, and Table 1a, Sample Grapheme Clusters, in Unicode Standard Annex #29, Unicode Text Segmentation.

use icu::segmenter::GraphemeClusterSegmenter;
let segmenter =
    GraphemeClusterSegmenter::new();

// நி (TAMIL LETTER NA, TAMIL VOWEL SIGN I) is an extended grapheme cluster,
// but not a legacy grapheme cluster.
let ni = "நி";
let egc_boundaries: Vec<usize> = segmenter.segment_str(ni).collect();
assert_eq!(&egc_boundaries, &[0, ni.len()]);

Implementations

impl GraphemeClusterSegmenter

pub const fn new() -> GraphemeClusterSegmenterBorrowed<'static>

Constructs a GraphemeClusterSegmenterBorrowed with an invariant locale from compiled data.

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

pub fn try_new_with_buffer_provider( provider: &(impl BufferProvider + ?Sized), ) -> Result<GraphemeClusterSegmenter, DataError>

A version of [Self :: new] that uses custom data provided by a BufferProvider.

✨ Enabled with the serde feature.

📚 Help choosing a constructor

pub fn try_new_unstable<D>( provider: &D, ) -> Result<GraphemeClusterSegmenter, DataError>

where D: DataProvider<SegmenterBreakGraphemeClusterV1> + ?Sized,

A version of Self::new that uses custom data provided by a DataProvider.

📚 Help choosing a constructor

⚠️ The bounds on provider may change over time, including in SemVer minor releases.

pub fn as_borrowed(&self) -> GraphemeClusterSegmenterBorrowed<'_>

Constructs a borrowed version of this type for more efficient querying.

Most useful methods for segmentation are on this type.

Trait Implementations

impl Debug for GraphemeClusterSegmenter

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.