ProtoBuff Data Types Schema Parser

A schema parser for Protocol Buffers data types. It plugs into @asyncapi/parser and lets you embed .proto message definitions directly in the payload (or schema) of an AsyncAPI document. During parsing, the Protobuf definition is converted into an AsyncAPI (JSON Schema based) schema so the rest of the AsyncAPI tooling can work with it like any other message payload.

Overview

AsyncAPI messages can carry payloads described in different schema languages. @asyncapi/parser delegates any payload whose schemaFormat it does not understand natively to a registered schema parser. This package is the schema parser for Protocol Buffers: you register it once, and every message payload tagged with a Protobuf schemaFormat is parsed and expanded in place into an equivalent AsyncAPI schema.

• Works with Protobuf 2 and Protobuf 3 schemas.
• Registers for the schema formats application/vnd.google.protobuf;version=2 and application/vnd.google.protobuf;version=3.
• Maps Protobuf messages, scalars, enums, repeated fields, oneof and nested types to JSON Schema.
• Understands leading comments as descriptions and a set of @-annotations for validation, defaults and examples.
• Optionally maps protoc-gen-validate and protovalidate rules to JSON Schema validators.
• Browser-compatible and ships both CommonJS and ES module builds.

There is no strict distinction between Protobuf 2 and 3. Declaring schemaFormat as application/vnd.google.protobuf;version=2 while providing a proto3 schema (or vice versa) does not by itself cause an error.

Version >= 2.0.0 of this package requires @asyncapi/parser >= 2.0.0; the 3.x line targets @asyncapi/parser >= 3.6.0.

• Installation
• Usage
• Supported input
• How it works
• Scalar type formats
• Comments and annotations
  • Per field annotation
  • Per message annotation
  • Head annotation
  • Head annotation "Option"
• Supported validation frameworks
• Development
• Contributing
• License

Installation

npm install @asyncapi/protobuf-schema-parser
# or
yarn add @asyncapi/protobuf-schema-parser

@asyncapi/parser is required to use this package. Requires Node.js >= 18.

Usage

Create a Parser, register the Protobuf schema parser on it, then parse an AsyncAPI document that uses a Protobuf schemaFormat. The Protobuf source goes into the message payload as a string.

import { Parser } from '@asyncapi/parser';
import { ProtoBuffSchemaParser } from '@asyncapi/protobuf-schema-parser';

const parser = new Parser();
parser.registerSchemaParser(ProtoBuffSchemaParser());

const asyncapiWithProto = `
asyncapi: 2.0.0
info:
  title: Example with ProtoBuff
  version: 0.1.0
channels:
  example:
    publish:
      message:
        schemaFormat: 'application/vnd.google.protobuf;version=3'
        payload: |
          message Point {
            required int32 x = 1;
            required int32 y = 2;
            optional string label = 3;
          }

          message Line {
            required Point start = 1;
            required Point end = 2;
            optional string label = 3;
          }
`;

const { document, diagnostics } = await parser.parse(asyncapiWithProto);

The same works with CommonJS:

const { Parser } = require('@asyncapi/parser');
const { ProtoBuffSchemaParser } = require('@asyncapi/protobuf-schema-parser');

const parser = new Parser();
parser.registerSchemaParser(ProtoBuffSchemaParser());

Notes:

• ProtoBuffSchemaParser is exported both as a named export and as the default export. Call it to obtain the parser instance, then pass that instance to registerSchemaParser.
• Place your Protobuf schema as a string in the message payload (AsyncAPI 2.x) or in the schema object's schema keyword (AsyncAPI 3.x) to get it parsed. See the fixtures under test/documents for both variants.
• After parsing, the message payload in document is the converted AsyncAPI schema.

Supported input

• Protobuf 2 and Protobuf 3 message definitions.
• The two registered schema formats:
  • application/vnd.google.protobuf;version=2
  • application/vnd.google.protobuf;version=3

References are not supported:

• No support for $ref inside the Protobuf payload.
• No support for import, with the exception of the bundled well-known definitions:
  • google/protobuf/* (provided by protobufjs)
  • google/type/* (bundled with this package)
  • the validation definitions validate/validate.proto and buf/validate/validate.proto

Any other import throws an error during parsing.

How it works

Each AsyncAPI document may contain several message payloads. For every payload whose schemaFormat matches one of the registered MIME types, the parser hands the raw Protobuf string to the converter, which uses protobufjs to parse it and then walks the message tree, emitting a JSON Schema based AsyncAPI schema.

Because a JSON Schema has a single root, the converter first determines the root message: the message that is not referenced as a field type by any other message. If several candidates remain, annotate the intended root with @RootNode; otherwise parsing fails with Found more than one root proto messages. If no root can be found, parsing fails with Not found a root proto messages.

The main mapping rules:

| Protobuf construct | AsyncAPI / JSON Schema output | |--------------------|-------------------------------| | message | { "title": <name>, "type": "object", "properties": { ... } }, plus a required list | | Scalar field (e.g. int32, string) | type + format from the scalar type map, plus x-primitive (and minimum/maximum for numeric types) | | Message-typed field | The referenced message compiled inline, plus x-type set to the Protobuf type name | | enum | { "type": "string", "enum": [<names>], "x-enum-mapping": { <name>: <number> } } | | repeated field | { "type": "array", "items": <field schema> } | | oneof with 2+ members | A property named after the oneof holding { "oneOf": [ ... ] }; each variant carries x-oneof-item with the field name | | Field / message comment | description (with @-annotations stripped out) |

Field membership in the required list is additive: a field is marked required if it is proto2 required, a non-optional proto3 field, or annotated with @Required.

Recursive messages are supported: when the same message type is encountered twice on the current branch, the converter stops descending to avoid an infinite loop.

Non-standard (x-) keywords are used to preserve Protobuf information that has no direct JSON Schema equivalent: x-primitive (original scalar type), x-type (referenced message/enum name), x-enum-mapping (enum name to numeric value), and x-oneof-item (the oneof field a variant came from).

Scalar type formats

Each Protobuf scalar type is mapped to a JSON Schema type plus a format that preserves the original Protobuf wire type. For example int64 becomes {"type": "integer", "format": "int64"} and bytes becomes {"type": "string", "format": "bytes"}. The Protobuf type is additionally kept in the non-standard x-primitive keyword. By default numeric types also carry minimum/maximum limits (see the primitiveTypesWithLimits option).

Comments and annotations

Each field of a message may have a comment which is reflected as the JSON Schema description. Furthermore, the comment can contain the following annotations:

message Point {
    /*
     * The coordinate on the x axis.
     * @Default 99
     * @Min 0
     * @Max 100
     */
    required int32 x = 1;

    /*
     * The coordinate on the y axis.
     * @Default 12
     * @Min 0
     * @Max 100
     */
    required int32 y = 2;
    optional string label = 3;
}

Per field annotation

| annotation | description | |------------|:------------| | @Example | JSON Schema examples keyword. Can appear multiple times. If used with a complex type, a single-line JSON object has to be used. | | @Min or @Minimum | JSON Schema numeric validator | | @Max or @Maximum | JSON Schema numeric validator | | @Pattern | JSON Schema string validator | | @ExclusiveMinimum | JSON Schema numeric validator | | @ExclusiveMaximum | JSON Schema numeric validator | | @MultipleOf | JSON Schema numeric validator | | @MinLength | JSON Schema string validator | | @MaxLength | JSON Schema string validator | | @MinItems | JSON Schema array validator | | @MaxItems | JSON Schema array validator | | @Default | JSON Schema default value | | @Required | Adds the field to the JSON Schema required list. Additive: a field is required if it is proto2 required, a non-optional proto3 field, or annotated with @Required. |

Per message annotation

| annotation | description | |------------|:------------| | @RootNode | If there are multiple types without a parent, you can give a hint about the root node with this annotation. |

Head annotation

| annotation | description | |------------|:------------| | @Option | In the head of your file you can place options for the parser. |

Head annotation "Option"

The @Option has to be followed by a space-separated option key and a space-separated value.

// @Option primitiveTypesWithLimits false

message Point {

}

Possible options are:

| option | description | default | |--------|:------------|:--------| | primitiveTypesWithLimits | If you do not want default minimum/maximum limits for primitive types, set this option to false. | true |

Supported validation frameworks

If you would like to add additional validation to your proto files, you can use one of the following validation frameworks. Their rules are translated into JSON Schema validation keywords during parsing.

• protoc-gen-validate — validation of lists is not 100% supported.
• protovalidate

Development

This project is written in TypeScript and builds both an ES module (esm/) and a CommonJS (cjs/) output. Tests use Jest against snapshot-style fixtures in test/documents.

npm install          # install dependencies

npm run build        # build both esm/ and cjs/ (build:esm, build:cjs)
npm test             # run the Jest test suite with coverage
npm run lint         # run ESLint (use lint:fix to auto-fix)

Source layout:

| Path | Purpose | |------|---------| | src/index.ts | Parser factory; implements the @asyncapi/parser schema-parser contract (parse, validate, getMimeTypes). | | src/protoj2jsonSchema.ts | Core converter that turns a Protobuf schema into an AsyncAPI schema. | | src/primitive-types.ts | Scalar Protobuf type to JSON Schema type/format map. | | src/google-types.ts | Bundled google/type/* well-known definitions. | | src/pathUtils.ts | Import path resolution helpers. | | src/protoc-gen-validate.ts | Maps (validate.rules) options to JSON Schema validators. | | src/protovalidate.ts | Maps (buf.validate.field) options to JSON Schema validators. | | test/documents/*.yaml | AsyncAPI input fixtures; *.result.json are the expected parsed outputs. |

Contributing

Read CONTRIBUTING to learn about the contribution process and the AsyncAPI Code of Conduct. Issues and pull requests are welcome.

License

Apache-2.0