@thanh01.pmt/research-kit
v1.8.0
Published
An autonomous Deep Research, Academic Harvest & Evidence Verification Engine for Curriculum OS & AI Agents.
Readme
@thanh01.pmt/research-kit
Autonomous Deep Research, Multimodal Academic Harvester & Pedagogical Evidence Verification Engine
Designed for Curriculum OS, AI Agents, and Intelligent Educational Authoring Platforms.
🎯 Overview
@thanh01.pmt/research-kit is an enterprise-grade research engine engineered to provide ground truth technical and pedagogical evidence before generating curricula, lesson plans, or coding projects.
It solves the fundamental problems of hallucinations, obsolete technical syntax, and lack of real-world curriculum grounding by orchestrating multi-engine web search, video transcript extraction, peer-reviewed academic literature discovery, bilingual terminology extraction, and natural language inference (NLI) claim verification.
Key Architectural Invariants
- Zero LLM Lock-in: Works 100% offline or with zero-token deterministic fallbacks by default. Accepts any injected LLM completion callback (
llm: LlmTextFn) from your host app (e.g. OpenAI, Anthropic, Gemini, OpenRouter, NineRouter, local vLLM). - Host Application API Key Injection: Host apps pass their own user/vault API keys (
tavily,exa,jina,serper,semanticScholar) without forcing keys into local package.envfiles. - Bilingual English & Vietnamese First-Class Support: Accurately extracts multi-syllable Vietnamese compound technical nouns (cảm biến siêu âm, điều biến độ rộng xung, mạch phân áp) paired with international canonical English terms.
- Dual Taxonomy Grounding: Maps topics simultaneously to ACM/IEEE CS2023 knowledge areas and CSTA K-12 Computer Science strands.
📦 The 6 Core Services
@thanh01.pmt/research-kit exposes a unified facade (ResearchKitClient) or standalone service classes:
┌───────────────────────────┐
│ ResearchKitClient │
└─────────────┬─────────────┘
│
┌───────────────┬───────────────┬───────────┼───────────┬───────────────┬───────────────┐
▼ ▼ ▼ ▼ ▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────────┐ ┌───────────────┐
│ Service 1 │ │ Service 2 │ │ Service 3 │ │ Service 4 │ │ Service 5 │ │ Service 6 │
│ Discovery │ │ Term & │ │ Academic │ │ Evidence │ │ Pedagogical │ │ Technical │
│ & Scrape │ │ Clean DAG │ │ Standards │ │ & NLI │ │ Synthesis │ │ Grounding │
│ Ingestion │ │ Extraction│ │ Alignment │ │ Auditor │ │ Orchestrator │ │ Factsheet │
└───────────┘ └───────────┘ └───────────┘ └───────────┘ └───────────────┘ └───────────────┘1. KnowledgeDiscoveryService (Discovery & Multimodal Ingestion)
- Multi-Engine Search: Auto-routes queries across Tavily, Exa, Serper, SearXNG, DuckDuckGo, and custom adapters.
- Commercial Noise Filter & Quality Auditor: Automatically identifies and rejects e-commerce clutter (Shopee, Lazada, Tiki, AliExpress, eBay, Amazon product listings) and calculates substantive technical density (code blocks, technical headings, tables).
- High-Fidelity Scraper: Jina Reader API / Crawl4AI / Direct Fetch extracting clean Markdown, formatted code snippets with correct language tags (
lua,cpp,python,swift), and tabular structures. - YouTube Transcript Ingestion: Parses video transcripts, stitches caption fragments into timestamped paragraphs (every ~30–45s), and filters out sponsor segments and subscribe filler noise.
2. TerminologyGraphService (Clean Prerequisite DAG & Concept Facets)
- Deterministic DAG & Prerequisite Engine: Applies transitive reduction to strip redundant shortcut edges ($A \to B \to C \implies$ remove $A \to C$) and runs Kahn's topological sort with cycle detection to guarantee clean, acyclic dependency graphs.
- Multi-dimensional Concept Facets: Concept extraction captures cognitive facets across Marr's tripartite layers:
facets.ulo: Conceptual purpose, real-world rationale & scientific nature.facets.cio: Algorithmic mental model, data flow & internal mechanics.facets.sio: Syntactic implementation, pinout connections & code drill.commonMisconceptions: Empirical student pitfalls and misconceptions.troubleshootingHints: Practical debugging and setup fixes.
- Tier 1 (Deterministic Candidate Harvester): Rapidly mines candidate terms from markdown headings, inline code blocks, bold markers, bilingual parenthetical pairs
Thuật ngữ (English), and Vietnamese technical head nouns. - Tier 2 (LLM Semantic Curator & Ontologist): Deduplicates synonyms (e.g., subsumes LDR, Quang trở, Cảm biến ánh sáng into
C_PHOTORESISTOR), assigns Marr cognitive layers (ULO/CIO/SIO) and Bloom taxonomy levels, and constructs genuine pedagogical prerequisite DAG edges (CONCEPT_DAG.tsv).
3. AcademicStandardsService (Academic Literature & Curricular Standards)
- Academic Search: Queries Semantic Scholar, OpenAlex (with automatic inverted index abstract reconstruction), and ArXiv.
- Dual Standards Mapping: Aligns computing topics with IEEE/ACM CS2023 (e.g.
PDC,AI,AR,HCI,GIT,SPD,FPL) and CSTA K-12 Strands (CS,NI,DA,AP,IC). - SOTA Comparison Matrix: Generates literature reviews and dynamic BibTeX citations (
REFERENCES.bib).
4. EvidenceVerificationService (Evidence Scoring, Fact-Checking & Provenance)
- Health Auditor: Rejects HTTP error pages, Cloudflare CAPTCHAs, paywalls, and low-substance spam.
- Structural & Pedagogical Scoring: Evaluates format richness, code presence, and noise factors (with optional LLM pedagogical scoring).
- Semantic Claim Verifier (Natural Language Inference): Evaluates whether claims are strictly
supported, directlyrefuted, or havenot_enough_evidenceagainst source text, detecting negation and contradictions. - Provenance Audit Trail: Emits immutable
PROVENANCE.mdandPROVENANCE.jsontracking source URLs, byte sizes, and access timestamps.
5. PedagogicalSynthesisService (Master Plan & Reference Pack Orchestration)
- Master Campaign Planner: Generates multi-angle search queries tailored to learner age band, hardware platform, and toolchains.
- URL Normalization & Deduplication: Strips tracking parameters (
utm_*,fbclid,gclid,#hash) to eliminate redundant crawling. - Canonical Reference Pack: Bundles all verified evidence into
REFERENCE_PACK.md,RESOURCE_CATALOG.md,KNOWLEDGE_GRAPH.md, andRESOURCE_INDEX.json.
6. TechnicalGroundingService (Hardware/Software Compatibility & Empirical Exemplars)
- Compatibility Matrix: Validates firmware, OS versions, libraries, and pinout requirements.
- Age Feasibility Constraints: Verifies if cognitive and motor skills match target age groups (e.g., fine soldering vs. breadboard dupont wires).
- Empirical Exemplar Projects: Extracts verified hands-on student projects with concrete hardware lists, target outcomes, and troubleshooting gotchas into
TECHNICAL_FACTSHEET.md.
🚀 Quick Start
Installation
pnpm add @thanh01.pmt/research-kit
# or
npm install @thanh01.pmt/research-kitBasic Usage with Host-Injected Keys & LLM
import { ResearchKitClient } from '@thanh01.pmt/research-kit';
// 1. Initialize client with host application settings
const client = new ResearchKitClient({
apiKeys: {
tavily: process.env.TAVILY_API_KEY,
jina: process.env.JINA_API_KEY,
exa: process.env.EXA_API_KEY,
serper: process.env.SERPER_API_KEY,
semanticScholar: process.env.SEMANTIC_SCHOLAR_KEY,
},
// Set to false in strict multi-tenant apps to prevent reading process.env
allowEnvFallback: false,
// Course context anchors search queries and cognitive age bands
defaultCourseContext: {
ageBand: [10, 14],
hardwarePlatform: 'Arduino Uno',
softwareEnv: 'mBlock 5 & Arduino IDE',
inScopeTopics: ['Cảm biến ánh sáng LDR', 'Mạch phân áp', 'Đèn LED PWM'],
},
// Pluggable LLM completion callback from your app runtime
llm: async (prompt, options) => {
const response = await fetch('https://openrouter.ai/api/v1/chat/completions', {
method: 'POST',
headers: {
'Authorization': `Bearer ${process.env.OPENROUTER_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'google/gemini-2.0-flash-001',
temperature: options?.temperature ?? 0.2,
messages: [
...(options?.system ? [{ role: 'system', content: options.system }] : []),
{ role: 'user', content: prompt },
],
}),
});
const data = await response.json();
return data.choices[0].message.content;
},
});
// 2. Execute Full Autonomous Research & Harvest
const result = await client.orchestrator.runFullHarvest('Nhà Thông Minh với Arduino', {
isDeep: true,
onProgress: (stage, message) => console.log(`[${stage}] ${message}`),
});
console.log('Distilled Concepts:', result.concepts.length);
console.log('Concept DAG (TSV):\n', result.catalog.conceptDagTsv);
console.log('Reference Pack:\n', result.referencePack.referencePackMd);🛠️ Specialized Service Examples
1. Bilingual Terminology & Prerequisite DAG Extraction
// Deterministic Tier 1 + LLM Tier 2 Semantic Curation
const termBundle = await client.terminology.extractConcepts(
'Cảm biến ánh sáng LDR với Arduino',
scrapedMarkdownText
);
for (const c of termBundle.concepts) {
console.log(`[${c.id}] ${c.termVi} (${c.termEn})`);
console.log(` Marr Layer: ${c.marrLayer} | Bloom: ${c.bloomLevel}`);
console.log(` Prerequisites: [${c.prerequisites.join(', ')}]`);
}2. Semantic Claim Verification (NLI Fact-Checking)
const claims = [
'Cảm biến ánh sáng LDR thay đổi điện trở khi cường độ ánh sáng thay đổi.',
'Cảm biến LDR tạo ra tín hiệu số truyền thẳng qua Bluetooth mà không cần điện áp phân áp.',
];
const report = await client.verification.verifyClaims(claims, referenceDocumentMarkdown);
for (const c of report.claims) {
console.log(`Claim: "${c.claimText}"`);
console.log(`Verdict: ${c.verdict.toUpperCase()} (${c.confidenceScore}%)`);
console.log(`Evidence Quote: "${c.extractedQuote}"`);
console.log(`Explanation: ${c.explanation}`);
}3. YouTube Transcript Ingestion
const ytResult = await client.discovery.ingestYouTube(
'https://www.youtube.com/watch?v=VIDEO_ID',
['vi', 'en']
);
console.log('Video Title:', ytResult.title);
console.log('Clean Transcript Paragraphs:\n', ytResult.stitchedMarkdown);🧪 Testing & Verification
Run the comprehensive unit test suite:
pnpm testTypecheck and build package distributions:
pnpm typecheck
pnpm build📄 License
MIT © Thanh Pham Part of the LearnWell Platform & Curriculum OS initiative.
