@diplodoc/sentenizer

Rule-based NLP library for sentence segmentation with Russian language support. Splits text into sentences using hand-crafted rules that handle Russian-specific cases like abbreviations, initials, and punctuation.

Features

• Rule-based segmentation — Hand-crafted rules optimized for Russian text
• Russian-specific handling — Correctly handles abbreviations (e.g., "и т. д.", "т. п."), initials (e.g., "И. В. Иванов"), quotations, and brackets
• Functional programming — Uses Ramda for clean, composable code
• High accuracy — Tested on various Russian text samples
• Lightweight — No external NLP libraries, fast and self-contained

Installation

npm install @diplodoc/sentenizer

Usage

Basic Example

const {sentenize} = require('@diplodoc/sentenizer');

const text = 'Он купил фрукты - яблоки, бананы, и т. д. все были очень рады угощению. Вот такой он добродушный наш родственник И. В. Иванов.';

const sentences = sentenize(text);
// sentences:
// [
//  'Он купил фрукты - яблоки, бананы, и т. д. все были очень рады угощению.',
//  'Вот такой он добродушный наш родственник И. В. Иванов.'
// ]

ES Modules

import {sentenize} from '@diplodoc/sentenizer';

const sentences = sentenize('Первое предложение. Второе предложение!');
// ['Первое предложение.', ' Второе предложение!']

API

sentenize(text: string): string[]

Splits text into sentences.

Parameters:

• text (string) — Text to segment

Returns:

• string[] — Array of segmented sentences

Type signature:

sentenize :: string -> string[]

How It Works

The library uses a rule-based approach with two types of conditions:

Join Conditions (chunks should be merged):

• Abbreviations (e.g., "и т. д.", "т. п.")
• Initials (e.g., "И. В. Иванов")
• Quotations and brackets
• Lowercase continuation
• Delimiters and spaces

Break Conditions (chunks should be split):

• Hard breaks (double newlines)
• Uppercase start after sentence end

Integration

This package is used by @diplodoc/translation for text segmentation during translation.

Development

See AGENTS.md for detailed development guidelines.

License

MIT