NLdoc OpenAPI Test Set Generator

A powerful CLI tool for generating comprehensive test sets from OpenAPI 3.1.0 specifications. This TypeScript library automatically creates valid and invalid test examples based on schema definitions, enabling robust API validation testing.

Overview

The OpenAPI Test Set Generator extracts examples from OpenAPI specifications and systematically creates test variants to validate API implementations. It processes schema definitions and generates:

• Valid examples: Test cases that conform to the API specification
• Invalid examples: Test cases that deliberately violate schema constraints for negative testing
• Property variants: Multiple variations of examples with modified properties based on configurable rules

This tool is essential for:

• Automated API testing and validation
• Schema compliance verification
• Test-driven API development
• Generating comprehensive test suites from API specifications

Getting Started

Prerequisites

• Node.js >= 22.0.0
• npm >= 10.0.0

Installation

Install the package globally:

npm install -g @nldoc/openapi-test-set-generator

Or as a development dependency:

npm install --save-dev @nldoc/openapi-test-set-generator

Basic Usage

Create a configuration file (e.g., test-generator.yaml):

input:
  openapi: ./api/openapi.json
output:
  valid: ./tests/examples.valid.json
  invalid: ./tests/examples.invalid.json
variantsOnExamples:
  property:
    # All examples in openapi.json are valid by default
    - where: {}
      do: {}
      valid: true
    
    # Generate invalid examples by removing required properties
    - where:
        required: true
        hasDefault: false
      do:
        remove: true
      valid: false

Run the generator:

openapi-test-set-generator extract-examples --input test-generator.yaml

Or using npx:

npx @nldoc/openapi-test-set-generator extract-examples -i test-generator.yaml

Configuration

Configuration File Structure

The generator uses a YAML configuration file with the following structure:

input:
  openapi: <path-to-openapi-spec>  # Path to your OpenAPI specification (JSON/YAML)

output:
  valid: <output-path-for-valid-examples>      # Where to save valid test examples
  invalid: <output-path-for-invalid-examples>  # Where to save invalid test examples

variantsOnExamples:
  property:
    - where: <conditions>  # Conditions to match properties
      do: <action>         # Action to perform on matched properties
      valid: <boolean>     # Whether the result should be valid or invalid

Property Modification Rules

The generator supports sophisticated property selection and modification:

Where Conditions

Select properties based on their schema characteristics:

where:
  type: 'string'           # Match by type
  format: 'email'          # Match by format
  required: true           # Match required properties
  hasDefault: false        # Match properties without defaults
  minLength:
    $gte: 1                # Match strings with minLength >= 1
  maxLength:
    $lte: 100              # Match strings with maxLength <= 100

Modification Actions

do:
  set: <value>    # Set property to a specific value
  remove: true    # Remove the property entirely
  truncate: 10    # Truncate string to specified length

Nested Property Support

The generator automatically traverses nested object structures to find and modify properties at any depth:

variantsOnExamples:
  property:
    # Modifies all email fields, even deeply nested ones
    - where:
        type: 'string'
        format: 'email'
      do:
        set: 'invalid-email'
      valid: false
    
    # Removes all non-required nested properties
    - where:
        required: false
      do:
        remove: true
      valid: true

Development

Local Setup

1. Clone the repository:

   git clone https://gitlab.com/logius/nldoc/lib/typescript/openapi-test-set-generator.git
   cd openapi-test-set-generator

2. Install dependencies:

   npm install

3. Build the project:

   npm run build

Available Scripts

• npm run build - Compile TypeScript to JavaScript
• npm run build:check - Type-check without emitting files
• npm test - Run test suite with coverage
• npm run test:watch - Run tests in watch mode
• npm run lint - Lint the codebase
• npm run format - Format code with Prettier
• npm run format:check - Check code formatting
• npm run fix - Auto-fix linting and formatting issues

API Documentation

CLI Commands

extract-examples

Extract and generate test examples from an OpenAPI specification.

openapi-test-set-generator extract-examples --input <config-file>

Options:

• -i, --input <file> - Path to the configuration file (required)

Examples:

openapi-test-set-generator extract-examples --input otsg.yaml
openapi-test-set-generator extract-examples -i otsg.yml

Programmatic Usage

import { 
  importOpenAPIFile,
  extractSchemaObjectsWithExamples,
  createExampleVariantsFromSchemasOnQueries
} from '@nldoc/openapi-test-set-generator';

// Load OpenAPI specification
const spec = await importOpenAPIFile('path/to/openapi.json');

// Extract schemas with examples
const schemas = extractSchemaObjectsWithExamples(spec);

// Generate test variants
const { valid, invalid } = await createExampleVariantsFromSchemasOnQueries(
  schemas,
  variantQueries
);

Example Scenarios

Common Test Patterns

1. Required Field Validation

   - where:
       required: true
       hasDefault: false
     do:
       remove: true
     valid: false

2. String Format Validation

   - where:
       type: 'string'
       format: 'uuid'
     do:
       set: 'invalid-uuid'
     valid: false

3. Array Constraints

   - where:
       type: 'array'
       required: true
       hasDefault: false
     do:
       set: []
     valid: false

4. Boolean Value Testing

   - where:
       type: 'boolean'
     do:
       set: true
     valid: true
   - where:
       type: 'boolean'
     do:
       set: false
     valid: true

Integration Examples

This generator can be integrated into any project that uses OpenAPI specifications. Common integration patterns include:

1. Generate comprehensive test sets for API schemas
2. Validate API implementation compliance
3. Create edge cases for robust testing
4. Ensure API specification accuracy

Example integration in package.json:

{
  "scripts": {
    "build:examples": "openapi-test-set-generator extract-examples -i ./example-generator.yaml"
  }
}

Testing

Run the test suite:

npm test

Run tests with coverage:

npm run test:watch run

Contributing

We welcome contributions! Please ensure:

1. All tests pass (npm test)
2. Code is properly formatted (npm run format:check)
3. Linting rules are followed (npm run lint)
4. TypeScript compilation succeeds (npm run build:check)

License

This project is licensed under the European Union Public License 1.2 - see LICENSE for details.

Acknowledgements

• Built with Oclif CLI framework
• Supports OpenAPI 3.1.0 specifications
• Uses Vitest for testing
• Part of the NLdoc ecosystem