reference-coverage-tester [WIP]

A tool to test the documentation and sample coverage for DocFX YML files. This tool is designed by and for the Office Platform content team (OP Content) to measure the effectiveness of the reference documentation in office-js-docs-reference and office-scripts-docs-reference.

Purpose

Currently, the primary purpose of this tool is to measure sample coverage. OP Content has the philosophy that developers prioritize API reference documentation and samples. By ensuring the two are found together, we provide higher educational potential.

The output is a .csv file with package-organized data that shows meaningful coverage against a well-defined success metric. In OP Content, we weight the API pages against their page views to determine high-value samples and sample opportunities.

Usage

Command Line Interface

reference-coverage-tester <config-file-path>

Parameters:

• <config-file-path>: Path to the JSON configuration file

Example:

reference-coverage-tester ./config.json

Configuration File

The tool requires a JSON configuration file with the following structure:

{
    "inputFolders": [
        "./path/to/yaml/files"
    ],
    "outputFolder": "./output",
    "notIncludedPatterns": [
        "*.interfaces.*",
        "temp-*",
        "**/draft/**"
    ]
}

Configuration Options:

• inputFolders: Array of directory paths containing DocFX YAML files to analyze. The tool will recursively scan all subdirectories for .yml and .yaml files.
• outputFolder: Directory where the CSV report will be generated (as "API Coverage Report.csv").
• notIncludedPatterns: Array of wildcard patterns to exclude files from analysis. Files whose paths match any of these patterns will be skipped. Supports * (matches any characters) and ? (matches single character) wildcards.

Wildcard Pattern Examples:

• *.interfaces.* - Excludes files with ".interfaces." anywhere in the filename
• temp-* - Excludes files starting with "temp-"
• **/draft/** - Excludes any files in "draft" directories at any level
• test?.yml - Excludes files like "test1.yml", "testA.yml", but not "test10.yml"

Note: The tool automatically excludes toc.yml files and invalid YAML files during processing.

Contributions and maintenance

Contributions are welcome. The current scope is one team and two repos. The documentation there has some level of customization, so improvements to this tool would need to keep that in mind.

Understanding the CSV file

Each row of the CSV has these values: Package,Class,Field,Type,Description Rating,Has Example?

• Package: The name of the package that contains the API element. For package-level entries, this shows the package name itself.
• Class: The name of class, interface, or enum. For package-level entries, this shows "N/A".
• Field: The name of the property, method, or enum field. "N/A" represents the row for the class or package description.
• Type: What category of API element this is (Package, Class, Interface, Enum, Method, Property, etc.).
• Description Rating: An automated evaluation of the JSDoc description quality. Could be "Missing", "Poor", "Fine", "Good", "Great", "Excellent". "Deprecated" is given if the @deprecated tag is found.
• Has Example?: A boolean of whether there is example code associated with this API.

Description Rating Logic

• Class/Interface/Package-level ratings (rows with "N/A" in the Field column): Based solely on the class's own summary description, not influenced by individual property or method descriptions.
• Property/Method/Field-level ratings: Based on the individual item's own description.

This separation ensures that a class with a good summary description gets a good rating even if some of its properties or methods have poor descriptions, and vice versa.