http-utils-processor-ts

RDF-Connect Typescript processors for handling HTTP operations. The main processor fetches a URL and writes the response to an output channel. It supports configurable HTTP methods, headers, authentication, and scheduling via cron expressions.

Usage

Installation

npm install
npm run build

Or install from NPM:

npm install @rdfc/http-utils-processor-ts

Pipeline Configuration Example

@prefix rdfc: <https://w3id.org/rdf-connect#>.
@prefix owl: <http://www.w3.org/2002/07/owl#>.

### Import the processor definitions
<> owl:imports <./node_modules/@rdfc/http-utils-processor-ts/processors.ttl>.

### Define the channels your processor needs
<out> a rdfc:Writer.

### Define and configure the processor
<fetcher> a rdfc:HttpFetch;
    rdfc:url "https://example.org/api/data";
    rdfc:writer <out>;
    rdfc:options [
        rdfc:method "GET";
        rdfc:headers "Authorization: Bearer TOKEN";
        rdfc:acceptStatusCodes "200-300";
        rdfc:closeOnEnd true;
        rdfc:timeOutMilliseconds 5000;
        rdfc:cron "*/5 * * * *";
        rdfc:runOnInit true;
        rdfc:errorsAreFatal true;
        rdfc:outputAsBuffer false;
        rdfc:auth [
            rdfc:type "basic";
            rdfc:username "user";
            rdfc:password "pass"
        ]
    ].

Configuration

Parameters of rdfc:HttpFetch:

• rdfc:url (string, required): URL(s) to fetch. Can be a single string or an array of strings.
• rdfc:writer (rdfc:Writer, required): Output channel to write the fetched response.
• rdfc:options (rdfc:HttpFetchOptions, optional): Optional settings including method, headers, timeout, authentication, cron, and more.

Parameters of rdfc:HttpFetchOptions:

• rdfc:method (string, optional): HTTP method (default: GET).
• rdfc:headers (string[], optional): Array of header strings (default: []).
• rdfc:acceptStatusCodes (string[], optional): List of accepted status codes or ranges, e.g., ["200", "201-300"] (default: ["200-300"]).
• rdfc:closeOnEnd (boolean, optional): Whether to close the writer after execution. Default depends on cron: true if no cron, false otherwise.
• rdfc:timeOutMilliseconds (integer, optional): Maximum wait time for a response before throwing a timeout error.
• rdfc:auth (rdfc:HttpFetchAuth, optional): Authentication configuration (see below).
• rdfc:cron (string, optional): Cron expression to schedule repeated executions.
• rdfc:runOnInit (boolean, optional): Run immediately upon initialization if cron is set (default: false).
• rdfc:errorsAreFatal (boolean, optional): Exit on fetch errors (default: true).
• rdfc:outputAsBuffer (boolean, optional): Whether the response is returned as a buffer (default: false).

Authentication (rdfc:HttpFetchAuth)

Supported types:

HTTP Basic Authentication

• rdfc:type: "basic"
• rdfc:username: Username string
• rdfc:password: Plaintext password

OAuth 2.0 Password Grant

• rdfc:type: "oauth2"
• rdfc:endpoint: URL of the OAuth 2.0 server
• rdfc:username: Username string
• rdfc:password: Plaintext password

Credentials are only sent to the authentication endpoint, not to the target URL.

Errors

All errors thrown are of type HttpFetchError and include a HttpUtilsErrorType enum describing the error nature.

Tests

Use Node.js to run tests:

npm run build
npm test

Some tests interact with real servers and may require credentials via a .env file:

# OAuth 2.0 Password Grant
RINF_USERNAME=
RINF_PASSWORD=

# HTTP Basic Auth
WoRMS_USERNAME=
WoRMS_PASSWORD=

# Set to true to enable real requests
BLUE_BIKE=true

Additional test information can be found here.