Why

Concordance shows, on every page, the pages related to it, and every link says where it comes from: written links always beat inferred ones. You install @concordance-wiki/concordance for that. This package is the part of it that produces the links: from the links people wrote, the references in the frontmatter, the mentions of a title in a section or a paragraph and the terms that keep appearing together; it combines their confidences, names the relation from the meta-model, keeps the neighbourhood a page shows, and recognises the documents that are twins of a note. Install it alone to build on the engine.

Quick start

npm install @concordance-wiki/inference

Two producers saw the same link; the build keeps one, at the combined confidence, with both provenances:

import { combineConfidences, combineLinks, combineOptions } from "@concordance-wiki/inference";
import { loadDefaultProfile } from "@concordance-wiki/profile";

combineConfidences([0.6, 0.5]); // 0.8, that is 1 − (1 − 0.6) × (1 − 0.5)
const links = [
  { from: "specs/link-broken", to: "glossary/finding", relation: "related", confidence: 0.6, provenance: [{ method: "section_mention", confidence: 0.6, path: "checks/link-broken.md", line: 12 }] },
  { from: "specs/link-broken", to: "glossary/finding", relation: "related", confidence: 0.5, provenance: [{ method: "explicit_link", confidence: 0.5, path: "checks/link-broken.md", line: 3 }] },
];
combineLinks(links, combineOptions(loadDefaultProfile())); // one link at 0.8, its two provenances in canonical order

What you get

• Four producers: explicitLinks, frontmatterLinks, mentionLinks, cooccurrenceLinks, from the written links, the reference attributes of the frontmatter, the occurrences of a title or alias, and the entities named in the same paragraph.
• One link, every proof: combineLinks keeps one link per source, target and relation, at 1 − Π(1 − cᵢ), with every provenance so the page can show why.
• A relation with a name: typeRelations and relationLabel read it from the meta-model (mapped section, typed attribute, single relation of a type pair, else related), with E-META-REL and I-REL-AMBIGUOUS when the notes disagree with it.
• The neighbourhood, precomputed: the bounded neighbours of every node and the ones a page displays, decided at build so the site needs no server.
• Twin resources: resolveDuplicateResources scores a document and a note that describe the same thing, MinHash and LSH on the text, and applies the decisions of the lock file.
• Contracts matched to notes: attachOperations joins every hand-written endpoint note to the operation imported from the contract of its API.
• References resolved: indexEntities and resolveReference read a frontmatter value by identifier, path or exact title.

Documentation

• Writing notes
• Configuration reference, the inference block
• Architecture
• Home page, the demo wiki and the changelog

Part of Concordance, GNU GPL v3 or later.

Explicit links (explicitLinks): every markdown link written in a note that resolves to another note gives a link at the explicit_link confidence of the profile, with the file, the line, the link text and the anchor as provenance; a link to a non-markdown file gives a documents link from the resource to the note; a missing target is E-LINK-BROKEN; a target in another source, written <source>:<path> or as a relative path climbing into a sibling source, resolves when inference.cross_source_links allows it and is W-LINK-CROSS-SOURCE otherwise. Links from one note to the same target are merged with every provenance kept; a link between two notes is related until the relation typing step names it. Bounded neighbourhood (accumulateCooccurrences, neighbourhoodOptions): two entities named in the same paragraph are neighbours, counted per paragraph; every node keeps its K best neighbours by count then by identifier (inference.neighbours.k, 50 by default), its row trimmed whenever it grows past 2K so that the full matrix is never held, and a paragraph naming more than 200 entities pairs only the first 200. cooccurrenceLinks turns every pair into one related link at the cooccurrence confidence with the number of shared paragraphs as provenance; neighbourhoodToModel writes the neighbours block of model.json. Confidence combination (combineLinks, combineOptions): one link per source, target, relation and attributes, at 1 − Π(1 − cᵢ) over its methods (combineConfidences, clamped to [0, 1] and rounded to four decimals), the glossary occurrences of a group counting as one method at the base of the strongest plus per_occurrence per additional occurrence up to cap (glossaryConfidence); every provenance is kept in canonical order, an exact duplicate once, and the function is pure and idempotent. Displayed neighbourhood (displayedNeighbourhood, displayOptions): the one-hop neighbours of every entity through the links of the model in either direction, typed entities and noteless keyword pages alike, each flagged by its kind, carrying the largest confidence and the relation of the most confident link between the two nodes, its direction seen from the entity and its rank, the position of its type in the display.neighbours_order the profile declares for the entity's type (types not listed, keyword pages included, share the rank after the last one); sorted by rank, then by decreasing confidence, then by identifier, and truncated after that to site.neighbourhood.size (6 by default, never more than MAX_DISPLAYED_NEIGHBOURS, 12), so that the nodes shown are the best of the priority order; a type without a declaration sorts by confidence alone; an entity without any neighbour has an empty list. displayedNeighbourhoodToModel writes the displayed_neighbourhood block of model.json, precomputed at build so that the browser never computes it.

Frontmatter references (frontmatterLinks): every reference-typed attribute of the profile that carries a relation (reads, writes, rules, roles, consumers, affects, broader, business_object, ...) turns each of its values into a link of that relation at the frontmatter_ref confidence, with the attributes the profile declares (accesses in read mode for reads), reversed when the attribute is inverse, and the file, line 1 and the attribute name as provenance. A value is resolved by identifier (full, or relative to the source of the note), then by source-relative path, then by exact title (resolveReference over an indexEntities index); a value that matches nothing, a title shared by several notes or a value that is not a string is W-REF-UNRESOLVED and gives no link; a note of a type the attribute does not accept gives its link, which the relation typing step drops with E-META-REL when the matrix forbids the relation. Several values naming the same target are merged with every provenance kept.

Relation typing (typeRelations): once every producer has spoken and their links are combined, names the relation of every link from the profile on four rungs, mapped section, typed frontmatter attribute, type pair admitting a single relation, related. The first two stand as produced and are only checked against the profile matrix: a relation the profile does not allow between the two types, or does not declare, drops the link with E-META-REL on the file, line and attribute of its provenance. A related link whose ordered type pair admits exactly one relation besides the wildcards takes it, marked relation_origin: pair in its attributes, turned around when only the reverse pair admits one; otherwise it stays related, capped at relations.related.cap (0.60) after the combination, and yields one I-REL-AMBIGUOUS finding on its first provenance that names a file; a related link known from co-occurrences alone yields none, since a co-occurrence is unnamed by nature and locates nothing. Links that then say the same thing are combined again, a pair-named link joining a declared link of the same triple and attributes and losing its marker; undirected relations go from the smaller identifier; a link with an endpoint the entity list does not type is kept as produced. relationLabel(profile, relation, locale, { inverse }) gives the label the site displays, label or inverse_label of the profile, never a string of the code. Both are pure, deterministic and, for the step, idempotent.

Mentions (mentionLinks): every occurrence of a note's title or alias read in another note gives a link. Under an H2 section that the profile maps for the type of the note (mappedSection: the heading is compared with the section key and with its label in every locale, ignoring case, accents and whitespace), the link carries the relation the section produces, its attributes and its direction (inverse sections link the mentioned entity to the note), at the section_mention confidence with the section key and the line as provenance; when the type pair does not admit that relation, or outside a mapped section, the mention gives a related link at the confidence the scan gave the occurrence (the base glossary_occurrence confidence, plus the type prefix bonus, halved for a homonym; the base alone when the occurrence carries none) with the line as provenance, oriented from the smaller identifier since related is undirected. When the occurrence carries its context, the provenance keeps it under occurrences (line, position, context, heading) with the matched text as written under text, so that the mentions panel of the site can show the passage and mark the words naming the entity. Self-mentions are dropped; mentions of one target in one relation merge into one link with a provenance per mention. The package declares the occurrence shape it reads structurally and does not depend on the scan. Twin resources (resolveDuplicateResources, duplicateOptions): every pair of resources is scored by adding its signals, capped at 1 (frontmatter declaration 1.0, same base name 0.7 in the same folder or 0.5 elsewhere, close base names at Jaro-Winkler 0.9 times 0.8, property title equal to the heading 0.6, similar text 0.7 from a Jaccard index of 0.8 or 0.4 from 0.6, same commit 0.3, folder proximity up to 0.2); above merge_above the pair merges into one group with every representation and the criterion that grouped it, from candidate_above it yields W-DUP-CANDIDATE. Text similarity runs on the words the caller provides through normalizeText (the comparison form of the language pack, stopwords removed): 5-word shingles, MinHash of 128 functions under a fixed seed, LSH banding so that only the pairs sharing a band are compared, exact Jaccard recomputed according to inference.duplicates.mode; the lock file's merged and separated pairs override the score; DuplicateStats counts the pairs brought together by the banding, the pairs scored, the exact verifications and the time spent, and formatDuplicateStats gives the build summary its lines.

Operation notes (attachOperations): every hand-written endpoint note is matched to an operation imported from the contract of its API, on the operation_id of its frontmatter, then on its method and path (or port and title for a SOAP operation), then on its title in comparison form with spaces and punctuation removed, against the operation title or identifier; the note names its API through the reference attribute of the profile (api), through a link already recorded with the API note, or is a candidate for every API of its source that has imported operations. A note stops at the first rung that matches, and an operation matched at a stronger rung is no longer offered to the weaker ones. A matched pair merges into the note, which keeps its identifier, markdown and frontmatter, takes the contract attributes it does not set, the operation's aliases and summary when it has none, lists the contract as a representation { kind: "contract", path, operation } next to its own file (after any twin resources already merged) and names the rung in grouped_by; the imported operation disappears and every link that named it, the exposes link of the API first of all, now names the note. A note matching several operations, or an operation claimed by several notes, yields W-OPERATION-AMBIGUOUS naming every candidate and attaches nothing; a note that names an API with an imported contract and matches none of its operations yields W-OPERATION-UNMATCHED, saying that the operation left the contract or that the note is ahead of it, while a note that names no API is left alone. The step is pure and deterministic and returns sorted entities, links and findings; in the pipeline it runs after the sources step, which imports the contracts, and before the mention scan, so that the merged note is the one the dictionary and the sections link.