sds-storage-writer-ts

TypeScript RDF-Connect processor for ingesting SDS streams and generating persistent Linked Data Event Stream (LDES) or in general TREE fragmentations.
The processor consumes an SDS stream and stores all data, metadata, and fragmentation structures into a supported storage backend.

Currently supported storage systems:

• MongoDB
• Redis

This repository exposes one processor:

rdfc:IngestSDS

Processor that ingests an SDS stream to generate a Linked Data Event Stream.
Uses sds:Bucket to describe fragmentation (see rdfc:Bucketize).

This processor stores:

• SDS members (data)
• SDS metadata
• SDS buckets and relations
• SDS fragmentation structure

in a persistent database backend, making the stream available for LDES servers to serve and traverse.

Import

@prefix owl: <http://www.w3.org/2002/07/owl#>.

<> owl:imports <./node_modules/@rdfc/sds-storage-writer-ts/processor.ttl>.

Example Configuration

@prefix rdfc: <https://w3id.org/rdf-connect#>.
@prefix owl:  <http://www.w3.org/2002/07/owl#>.

### Import processor definitions
<> owl:imports <./node_modules/@rdfc/sds-storage-writer-ts/processor.ttl>.

### Define readers
<dataInput> a rdfc:Reader.
<metadataInput> a rdfc:Reader.

### Configure the processor
<ingestor> a rdfc:IngestSDS;
    rdfc:dataInput <dataInput>;
    rdfc:metadataInput <metadataInput>;
    rdfc:database [
        rdfc:url "mongodb://localhost:27017/ldes";
        rdfc:metadata "meta";
        rdfc:data "data";
        rdfc:index "index";
    ].

Processor Semantics

Given an SDS stream with a predefined fragmentation:

# Member exists
ex:sample1 a ex:Object;
  ex:x "2";
  ex:y "5".

sds:DataDescription {
  [] sds:stream <bucketizedStream>;
    sds:payload ex:sample1; # Stream membership
    sds:bucket <bucket2>.

  # Bucket relations
  <bucket1> sds:relation [
    sds:relationType tree:GreaterThanRelation;
    sds:relationBucket <bucket2>;
    sds:relationValue 1;
    sds:relationPath ex:x 
  ].
}

The processor will:

• Store the member data ex:sample1
• Store SDS metadata sds:DataDescription and sds:RemoveDataDescription
• Store bucket definitions sds:bucket and sds:relationBucket values
• Store bucket relations sds:relation values
• Persist the full fragmentation structure in the configured storage system.

Database Backends

MongoDB

Data is stored in collections defined by:

• rdfc:metadata
• rdfc:data
• rdfc:index

Redis

Redis is supported as an alternative backend with the same logical structure, enabling:

• fast ingestion
• scalable storage

Configuration Reference

rdfc:IngestSDS

| Property | Type | Required | Description | |----------------------|---------------------|----------|-------------------------------| | rdfc:dataInput | rdfc:Reader | yes | SDS member stream | | rdfc:metadataInput | rdfc:Reader | yes | SDS metadata stream | | rdfc:database | rdfc:DatabaseConfig | yes | Storage backend configuration |

rdfc:DatabaseConfig

| Property | Type | Required | Description | |-----------------|--------|----------|--------------------------| | rdfc:metadata | string | yes | Metadata collection name | | rdfc:data | string | yes | Data collection name | | rdfc:index | string | yes | Index collection name | | rdfc:url | string | no | Database connection URL |

Role in RDF-Connect Pipelines

Typical architecture:

[SDS Stream] + [Members Stream]
↓
rdfc:IngestSDS
↓
MongoDB / Redis
↓
LDES Server

This processor acts as the persistent storage layer between streaming RDF-Connect pipelines and TREE/LDES serving infrastructure.

© Ghent University – IMEC
MIT License

Authors:

• Arthur Vercruysse
• Julián Rojas
• Ieben Smessaert