telo
v0.0.2
Published
Create, serialize, and gzip WARC 1.1 records from HTML or text content.
Readme
Telo is a compact library for creating, serializing, and compressing WARC (Web ARChive) records from HTML pages or text content. It supports WARC 1.1 and produces gzip-compressed output ready for storage or transfer.
Features
- Create WARC records (RESPONSE by default) with URL, HTTP headers, and custom metadata.
- Serialize WARCInfo records with metadata.
- Asynchronous content streaming for large pages.
- Gzip compression of combined WARCInfo and record.
- Fully compatible with WARC 1.1.
Installation
npm install teloUsage
import {gunzipSync} from 'node:zlib';
import {Headers} from 'http-directives';
const telo = new Telo();
const record = await telo.createRecord(
TeloTypes.RESPONSE,
'http://example.com',
'<body />',
{
[Headers.CONTENT_TYPE]: 'text/html; charset=UTF-8'
},
{
Browser: 'Firefox'
}
);
const uncompressed = gunzipSync(record);
const text = uncompressed.toString('utf-8');API
Telo.createRecord(type, url, page, httpHeaders?, info?);- type — WARC record type (default: RESPONSE)
- url — URL of the page
- page — HTML or text content
- httpHeaders — optional HTTP headers object
- info — optional metadata object
Returns: Promise<Buffer> — gzip-compressed WARC record.
Telo.serializeInfo(filename, info);- filename — WARC filename
- info — metadata object
Returns: Promise<Uint8Array> — serialized WARCInfo record.
License
MIT