hoard

Fetch web content and store it as markdown. Built to feed local search and RAG pipelines.

Each URL is fetched, stripped of noise, and written to a .hoard/ directory as a markdown file with YAML frontmatter. Re-running only refetches content that has changed.

Install

npm install -g @nickgarlis/hoard

Usage

Add a URL

hoard add https://pkg.go.dev/github.com/spf13/viper
hoard add https://pkg.go.dev/github.com/spf13/viper --tags config,golang

Creates hoard.json if it doesn't exist, fetches the URL immediately.

Fetch all sources

hoard fetch

Skips unchanged content (via ETags and content hashing). URLs removed from hoard.json have their files deleted.

List sources

hoard list

Remove a URL

hoard remove https://pkg.go.dev/github.com/spf13/viper

Options

All commands accept:

| Flag | Default | Description | |------|---------|-------------| | -c, --config <path> | hoard.json | Config file | | -o, --output-dir <path> | .hoard | Content store directory |

hoard fetch also accepts -f, --force to refetch everything regardless of cache.

hoard.json

{
  "sources": [
    {
      "url": "https://pkg.go.dev/github.com/spf13/viper",
      "tags": ["config", "golang"]
    }
  ]
}

Using with qmd

qmd is a local hybrid search engine for markdown files. hoard and qmd compose naturally — hoard writes .hoard/, qmd indexes it.

1. Fetch your sources:

hoard fetch

2. Register the store as a qmd collection (once):

qmd collection add .hoard --name hoard

3. Index and embed:

qmd update && qmd embed

4. Search:

qmd query "how do I configure viper"

After the initial setup, refresh with hoard fetch && qmd update && qmd embed whenever your sources change.