webnovel-scraper

A TypeScript library for extracting web novels — novel metadata, the ordered chapter list, and clean chapter text — from popular Vietnamese reading sites. It does extraction only: downloading, resuming, and exporting (EPUB/TXT/JSON) are separate layers built on top.

Package: @duyquangnvx/webnovel-scraper (public, published on npm). Status: pre-release (0.x — the public API is unstable and may change between minor versions). Extraction runs end-to-end for three built-in Vietnamese sources (truyenfull, dtruyen, sstruyen).

Setup

Built with Node 20+, TypeScript ESM, got-scraping + cheerio + zod, tested with vitest, bundled with tsup. Develop with pnpm install, then the scripts in CLAUDE.md.

npm install @duyquangnvx/webnovel-scraper

Usage

import { createScraper } from '@duyquangnvx/webnovel-scraper'

const scraper = createScraper()
const info = await scraper.getNovelInfo(url)          // NovelInfo
const chapters = await scraper.getChapterList(url)     // ChapterRef[] — ordered, paginated
const content = await scraper.getChapter(chapters[0])  // ChapterContent { title, paragraphs }

getChapter also accepts a bare chapter URL (getChapter(chapterUrl)); the source resolves it to a ChapterRef first. To check a URL before scraping, scraper.supports(url) returns a boolean without throwing, and scraper.supportedHosts lists the hostnames the registry accepts.

Whole-novel download, per-chapter error policy, and export are the consumer's job. The library hands back the ordered list; you drive the loop and decide what a failed chapter means. Always wrap each chapter in its own error boundary — one drifted or locked chapter must not abort the run (politeness is enforced per-host under the hood, so a tight loop stays safe):

import { createScraper, ChapterLockedError, ParseError } from '@duyquangnvx/webnovel-scraper'

const scraper = createScraper()
const chapters = await scraper.getChapterList(novelUrl)

for (const ref of chapters) {
  try {
    save(ref.order, await scraper.getChapter(ref))
  } catch (err) {
    if (err instanceof ChapterLockedError) continue // VIP/paywall — skip
    if (err instanceof ParseError) continue         // site drift — record & skip
    throw err                                        // FetchError / unexpected — surface or retry
  }
}

Development commands live in the table in CLAUDE.md.

Custom sources & transports

Built-in sources are produced by feeding a declarative SelectorConfig (per-site CSS selectors + a few typed URL functions) to createSelectorSource. The engine hides all shared machinery — fetch + per-host throttle/retry, pagination-follow, html-to-text content cleaning, zod validation. To support another static-selector site, pass your own config and register it:

import { createScraper, createSelectorSource, builtinSources } from '@duyquangnvx/webnovel-scraper'

const mySource = createSelectorSource(myConfig)
const scraper = createScraper({ sources: [...builtinSources, mySource] })

Transports are chosen per source by its renderingHint. createScraper takes a hint-keyed transports map (default: a built-in static HTTP transport); a source declaring renderingHint: 'dynamic' with no dynamic transport configured throws ConfigurationError at construction. Authentication is bring-your-own-session — supply per-host cookies/headers when building the HTTP transport; the library never logs in:

import { createScraper, createHttpFetcher } from '@duyquangnvx/webnovel-scraper'

const scraper = createScraper({
  transports: {
    static: createHttpFetcher({
      sessions: { 'site.example': { cookie: 'sid=…' } }
    })
  }
})

Structure

• src/core/ — Source interface, createScraper facade, registry, errors, schemas, boundary helpers (fetchOk, validate)
• src/fetch/ — the Fetcher transport seam: a got-scraping HTTP fetcher with per-host throttle + retry
• src/selector-source/ — the declarative createSelectorSource engine that builds a Source from a SelectorConfig
• src/sources/ — one config per site (truyenfull, dtruyen, sstruyen): a 4-line index.ts + a config.ts
• docs/ — requirements, architecture, decisions (ADRs), backlog

Target layout: docs/structure.md.

Releasing

Releases are automated by .github/workflows/release.yml: pushing a vX.Y.Z tag builds, publishes to npm with provenance, and opens a GitHub Release from the matching CHANGELOG.md section (see ADR-0007).

One-time setup: add an npm automation token as the NPM_TOKEN repository secret (Settings → Secrets and variables → Actions).

To cut a release:

1. In CHANGELOG.md, move the [Unreleased] entries under a new ## [X.Y.Z] - YYYY-MM-DD heading (leave a fresh empty [Unreleased] above), and commit.
2. npm version <patch|minor|major> — bumps package.json, commits, and creates the vX.Y.Z tag.
3. git push --follow-tags — CI publishes and creates the Release.

The first publish should be done manually (npm publish --access public) to claim the scoped name; wire the token and use the tag flow afterwards.

More

• Requirements: docs/requirements.md
• Architecture: docs/architecture.md
• Decisions (ADRs): docs/adr/
• Backlog: docs/backlog.md
• Agent config: CLAUDE.md