js-stream-rdata

js-stream-rdata reads R data frames from .RData and .rds files through the librdata C library.

The public TypeScript method signatures remain the same:

• getMetadata()
• getData()
• readRecords()
• getUniqueValues()

The metadata shape still follows the Dataset-JSON style used by the existing wrapper.

Supported formats

• .RData
• .rds

Features

• Read metadata from R data frames
• Read rows as arrays or keyed objects
• Stream records with async iteration
• Select columns
• Filter rows with js-array-filter
• Compute unique values per column

Installation

npm install js-stream-rdata

Build

Compile the TypeScript layer:

npm run build

Rebuild the local native addon:

npm run build:native

Generate packaged prebuild binaries:

npm run build:prebuilds

Usage

Generic reader

Use createDatasetReader() when you want the library to infer .RData vs .rds from the file extension.

import { createDatasetReader } from 'js-stream-rdata';

const dataset = createDatasetReader('/path/to/sample.RData');
const metadata = await dataset.getMetadata();
const { data, lastRow, endReached } = await dataset.getData({
  start: 0,
  length: 10,
});

Explicit reader

Use DatasetRData when the dataset is already known to be .RData or .rds.

import DatasetRData, { DatasetReader } from 'js-stream-rdata';

const dataset = new DatasetRData('/path/to/sample.rds');

const genericDataset = new DatasetReader('/path/to/sample.RData', {
  format: 'rdata',
});

Reading metadata

const metadata = await dataset.getMetadata();

Reading rows

const result = await dataset.getData({
  start: 0,
  length: 500,
  type: 'object',
  filterColumns: ['Name', 'Age'],
});

console.log(result.data);
console.log(result.lastRow);
console.log(result.endReached);

getData() returns:

{
  data: (ItemDataArray | ItemDataObject)[];
  lastRow: number;
  endReached: boolean;
}

Streaming rows

for await (const record of dataset.readRecords({
  start: 10,
  bufferLength: 1000,
  type: 'object',
  filterColumns: ['Name', 'Age'],
})) {
  console.log(record);
}

Filtering rows

Filtering is supported by getData() through js-array-filter.

import Filter from 'js-array-filter';
import { createDatasetReader } from 'js-stream-rdata';

const dataset = createDatasetReader('/path/to/sample.RData');
const metadata = await dataset.getMetadata();

const filter = new Filter('dataset-json1.1', metadata.columns, {
  conditions: [
    { variable: 'Age', operator: 'gt', value: 55 },
    { variable: 'Sex', operator: 'eq', value: 'F' },
  ],
  connectors: ['or'],
});

const filtered = await dataset.getData({
  start: 0,
  length: 100,
  type: 'object',
  filter,
  filterColumns: ['Name', 'Sex', 'Age'],
});

console.log(filtered.data);

A plain BasicFilter object can also be passed as filter.

Getting unique values

const uniqueValues = await dataset.getUniqueValues({
  columns: ['Name', 'Sex'],
  limit: 100,
  bufferLength: 1000,
  sort: true,
  addCount: true,
});

console.log(uniqueValues);

API summary

createDatasetReader(filePath, options?)

Creates a dataset reader by detecting the file format from the extension.

new DatasetReader(filePath, options?)

Creates the generic dataset reader.

Options:

• encoding?: BufferEncoding default utf8
• checkExists?: boolean default false
• format?: 'rdata' | 'rds'

getMetadata(forceReload?)

Returns Promise<DatasetMetadata>.

getData(props)

Parameters:

• start?: number
• length?: number use -1 to read to the end
• type?: 'array' | 'object'
• filterColumns?: string[]
• filter?: Filter | BasicFilter
• chunkSize?: number

Returns a promise resolving to an object with data, lastRow, and endReached.

readRecords(props?)

Parameters:

• start?: number
• bufferLength?: number
• type?: 'array' | 'object'
• filterColumns?: string[]

Returns AsyncGenerator<ItemDataArray | ItemDataObject, void, undefined>.

getUniqueValues(props)

Parameters:

• columns: string[]
• limit?: number
• bufferLength?: number
• sort?: boolean
• addCount?: boolean

Returns Promise<UniqueValues>.

Running tests

npm test

License

This project is licensed under the MIT License. See the LICENSE file for details.

Author

Dmitry Kolosov

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.

For more details, refer to the source code and the documentation.