lws-test-suite

v0.0.2

Published

3 days ago

W3C Linked Web Storage Protocol conformance test suite

Downloads

155

0High
0Medium
0Low

melvincarvalho

lws linked-web-storage w3c conformance test-suite protocol testing earl

W3C Linked Web Storage Protocol Test Suite

Official conformance test suite for the W3C Linked Web Storage (LWS) Protocol

Overview

This test suite validates conformance with the W3C Linked Web Storage Protocol specification. It provides automated testing, W3C EARL conformance reporting, and multi-implementation validation for LWS servers.

Status: MVP Release - Core MUST-level requirements (27 tests)

Maintainer: W3C LWS Working Group

Quick Start

# Install dependencies
npm install

# Test an LWS server implementation
npm test

# View conformance report
open reports/html/lws-server.html

Features

✅ W3C EARL 1.0 Reporting - Machine-readable RDF conformance reports
✅ Multi-Implementation Testing - Test multiple LWS servers with a single command
✅ Automated Server Management - Auto-start/stop servers during testing
✅ Spec Traceability - Every test linked to specification section
✅ Three Report Formats - EARL (RDF), HTML (human), JSON (CI/CD)
✅ Minimal Dependencies - Only 3 runtime dependencies (n3, commander, chalk)
✅ Standards Compliant - Follows W3C testing best practices

Test Coverage

Current Coverage (v0.0.1)

Total: 27 MUST-level conformance tests

| Category | Tests | Status | |----------|-------|--------| | HTTP Methods | 12 | ✅ GET, PUT, POST, DELETE, HEAD, OPTIONS | | Concurrency Control | 5 | ✅ ETags, If-Match, If-None-Match | | Containers | 4 | ✅ JSON-LD format, @context, contains | | HTTP Headers | 6 | ✅ Link, CORS, Location, Allow |

Specification Sections Covered

§4.1 - Resource Retrieval (GET, HEAD)
§4.2 - Resource Creation & Update (PUT)
§4.3 - Server-Assigned URIs (POST)
§4.4 - Resource Deletion (DELETE)
§4.5 - Metadata Operations (HEAD)
§4.6 - CORS Support (OPTIONS)
§5 - Concurrency Control (ETags, Conditional Headers)
§6 - Container Operations (Listings, JSON-LD)
§7 - HTTP Headers (Link, CORS, Location)

Conformance Levels

✅ MUST - 27 tests (100% coverage of core requirements)
⏳ SHOULD - Planned for v0.1.0
⏳ MAY - Planned for v0.2.0

Installation

Global Installation

npm install -g lws-test-suite
lws-test --subject lws-server

Local Installation

npm install lws-test-suite
npx lws-test --subject lws-server

From Source

git clone https://github.com/linkedwebstorage/test-suite
cd test-suite
npm install
npm test

Usage

Testing a Managed Server

For servers that can be started automatically:

# Test lws-server (auto-starts on port 3126)
npm test -- --subject lws-server

# Test JavaScriptSolidServer
npm test -- --subject jss

# Verbose output with detailed request/response logs
npm test -- --verbose

Testing an External Server

For already-running servers:

Create a configuration file my-server.config.json:

{
  "name": "my-lws-server",
  "version": "1.0.0",
  "type": "external",
  "server": {
    "healthCheck": {
      "url": "https://my-server.example.com/",
      "expectedStatus": 200
    }
  },
  "baseUrl": "https://my-server.example.com"
}

Run tests:

npm test -- --config my-server.config.json

Testing a Custom Managed Server

For servers you want the suite to start/stop:

{
  "name": "my-lws-server",
  "version": "1.0.0",
  "type": "managed",
  "server": {
    "command": "node",
    "args": ["./server.js", "--port", "3128"],
    "startupTimeout": 5000,
    "healthCheck": {
      "url": "http://localhost:3128/",
      "expectedStatus": 200
    }
  },
  "baseUrl": "http://localhost:3128",
  "cleanup": {
    "dataDirectory": "./test-data"
  }
}

CLI Options

lws-test [options]

Options:
  --subject <name>     Test subject name (default: "lws-server")
  --config <path>      Custom configuration file
  --level <level>      Filter by conformance level (MUST, SHOULD, MAY)
  --report <format>    Report format: earl, html, json, all (default: "all")
  --verbose            Detailed test output with HTTP logs
  -V, --version        Output version number
  -h, --help           Display help

Examples

# Test only MUST-level requirements
npm test -- --level MUST

# Generate only EARL report
npm test -- --report earl

# Test with verbose HTTP logging
npm test -- --verbose

# Test custom server
npm test -- --config ./servers/my-impl.config.json

Reports

The test suite generates three complementary report formats:

1. EARL Report (W3C Standard)

Format: RDF Turtle (conforming to W3C EARL 1.0 Schema)

Location: reports/earl/{server-name}.ttl

Use Case: Submitting conformance results to W3C working groups

Example:

@prefix earl: <http://www.w3.org/ns/earl#> .

[] a earl:Assertion ;
   earl:assertedBy <#test-harness> ;
   earl:subject <#lws-server> ;
   earl:test <https://w3c.github.io/lws-protocol/tests#test-get-resource> ;
   earl:result [
     a earl:TestResult ;
     earl:outcome earl:passed ;
     dc:date "2026-01-14T16:00:00Z"^^xsd:dateTime
   ] .

2. HTML Report (Human-Readable)

Format: Interactive HTML with CSS

Location: reports/html/{server-name}.html

Use Case: Visual review of test results, sharing with team

Features:

Color-coded pass/fail status
Grouped by test category
Pass rate visualization
Detailed error messages
Filterable results

3. JSON Report (Machine-Readable)

Format: Structured JSON

Location: reports/json/{server-name}.json

Use Case: CI/CD integration, automated processing

Example:

{
  "subject": {
    "name": "lws-server",
    "version": "0.0.2"
  },
  "testDate": "2026-01-14T16:00:00Z",
  "results": [
    {
      "testId": "test-get-resource",
      "name": "GET existing resource",
      "level": "MUST",
      "outcome": "passed",
      "duration": 9
    }
  ]
}

Architecture

Directory Structure

test-suite/
├── manifests/              # RDF Turtle test manifests
│   ├── manifest.ttl       # Root manifest (links all tests)
│   ├── http-methods-manifest.ttl
│   ├── etag-manifest.ttl
│   ├── containers-manifest.ttl
│   └── headers-manifest.ttl
│
├── tests/                  # JavaScript test implementations
│   ├── http-methods/      # GET, PUT, POST, DELETE, HEAD, OPTIONS
│   ├── etag/              # If-Match, If-None-Match
│   ├── containers/        # JSON-LD listings
│   └── headers/           # Link, CORS, Location
│
├── lib/                    # Test harness
│   ├── runner.js          # Test orchestration
│   ├── client.js          # HTTP test client
│   ├── server-manager.js  # Server lifecycle
│   ├── manifest-parser.js # RDF manifest parser
│   ├── earl-reporter.js   # EARL report generator
│   └── html-reporter.js   # HTML report generator
│
├── config/                 # Implementation configurations
│   ├── test-subjects.ttl  # RDF registry
│   ├── lws-server.config.json
│   └── jss.config.json
│
├── bin/                    # CLI entry point
│   └── lws-test.js
│
└── reports/                # Generated reports (gitignored)
    ├── earl/
    ├── html/
    └── json/

Test Format

Tests are JavaScript functions with metadata annotations:

/**
 * @test-id lws:test-get-resource
 * @spec-section 4.1.1
 * @spec-requirement MUST return 200 for existing resources
 * @level MUST
 */
export async function testGetResource(client) {
  // Create test resource
  await client.createResource('/test.json', { data: 'test' });

  // Execute request
  const response = await client.get('/test.json');

  // Assert requirements
  assert.strictEqual(response.status, 200);
  assert.ok(response.headers.get('ETag'));
}

Manifest Format

Tests are declared in RDF Turtle manifests:

<#test-get-resource>
  rdf:type lws:ProtocolTest ;
  mf:name "GET existing resource" ;
  rdfs:comment "Server MUST return 200 for existing resources" ;
  lws:specSection "4.1.1" ;
  lws:conformanceLevel "MUST" ;
  lws:category "HTTP Methods" ;
  lws:testImplementation "tests/http-methods/get.test.js#testGetResource" .

Tested Implementations

| Implementation | Version | Pass Rate | Report | Notes | |----------------|---------|-----------|--------|-------| | lws-server | 0.0.2 | 100% (27/27) | View | Full compliance | | JavaScriptSolidServer | 0.0.77 | 14.8% (4/27) | View | Requires authentication* |

Authentication Note: JavaScriptSolidServer requires authentication for all resource operations, even in --lws-mode. The test suite currently tests unauthenticated LWS protocol compliance. Only CORS/OPTIONS tests (4/27) pass without authentication. Future versions may add authentication support for testing authenticated LWS implementations.

Contributing

Adding New Tests

Create test file in appropriate category:

// tests/http-methods/patch.test.js
export async function testPatchResource(client) {
  // Test implementation
}

Add to manifest:

<#test-patch-resource>
  lws:testImplementation "tests/http-methods/patch.test.js#testPatchResource" .

Run tests to verify:
```
npm test
```

Test Guidelines

One assertion per test - Each test validates a single specification requirement
Clear naming - Use descriptive names: testGetResource, not test1
Metadata complete - Always include @test-id, @spec-section, @level
Cleanup resources - Use client.createResource() for automatic cleanup
Idempotent - Tests should not depend on execution order

Reporting Issues

Report bugs and feature requests at: https://github.com/linkedwebstorage/test-suite/issues

Roadmap

v0.1.0 (Q2 2026)

[ ] SHOULD-level tests (15+ additional tests)
[ ] Enhanced error messages with debugging hints
[ ] CI/CD integration examples (GitHub Actions)
[ ] Performance benchmarking suite

v0.2.0 (Q3 2026)

[ ] MAY-level tests (optional features)
[ ] Content negotiation tests (Turtle ↔ JSON-LD)
[ ] WebSocket notification tests
[ ] Batch testing (multiple implementations in one run)

v1.0.0 (Q4 2026)

[ ] Complete specification coverage
[ ] Authentication/Authorization tests
[ ] Web-based test runner UI
[ ] W3C Recommendation track alignment

Requirements

Node.js >= 18.0.0
npm >= 8.0.0

Dependencies

Runtime (3 total):

n3 (^1.17.2) - RDF Turtle parsing and generation
commander (^11.1.0) - CLI argument parsing
chalk (^5.3.0) - Terminal output formatting

Development:

lws-server (^0.0.2) - Reference LWS implementation for testing

Built-in (no install required):

node:assert - Test assertions
node:fs - File system operations
fetch - HTTP client (Node 18+)

License

W3C Software and Document License

This work is made available under the W3C Software and Document License, allowing free use, modification, and distribution with proper attribution.

Acknowledgments

Developed by: W3C LWS Working Group

Built for testing:

lws-server - Minimal LWS reference implementation
JavaScriptSolidServer - Full-featured LDP/LWS/Solid server

Special thanks to:

Solid Community for pioneering conformance testing approaches
W3C LDP Working Group for foundational specifications
Contributors to the RDF and Linked Data ecosystem

References

Specifications

W3C LWS Protocol - Linked Web Storage specification
W3C EARL 1.0 - Evaluation and Report Language
W3C LDP - Linked Data Platform

Related Projects

W3C Resources

Questions? Contact the W3C LWS Working Group

Contributing? See CONTRIBUTING.md for guidelines