host-consensus

http://example.com\@evil.com is example.com to JavaScript and evil.com to Python.

A runtime-path library: you import it into request-validation code and ship it. Zero runtime dependencies. TypeScript, ES2022, dual ESM/CJS, with a small CLI.

npm install host-consensus

| | reads the host as | |---|---| | new URL(), the WHATWG rules every browser and Node run | example.com | | urllib.parse, the RFC 3986 rules most of the non-browser world runs | evil.com |

Validate that URL in Node against an allowlist and it passes as example.com. Hand the same string to a Python service, a Go proxy, or anything else built on RFC 3986, and the request goes to evil.com.

Neither parser is broken. WHATWG URL says a backslash is a path separator in a special scheme; RFC 3986 says it is an ordinary character, so the authority runs to @ and the userinfo is example.com\. Both are behaving exactly as specified.

What it does

import { inspect, agrees, assertConsensus } from 'host-consensus';

agrees('https://example.com/a');            // true
agrees('http://example.com\\@evil.com');    // false

assertConsensus('http://example.com\\@evil.com');
// throws HostDisagreementError:
//   "http://example.com\@evil.com" is read as different hosts
//   (whatwg=example.com, rfc3986=evil.com)

inspect returns both readings so you can log what actually differed:

inspect('http://example.com\\@evil.com');

{
  "input": "http://example.com\\@evil.com",
  "verdict": "host-differs",
  "host": null,
  "readings": [
    { "convention": "whatwg",  "host": "example.com", "port": "", "origin": "http://example.com" },
    { "convention": "rfc3986", "host": "evil.com",    "port": "", "origin": "http://evil.com" }
  ]
}

From the command line:

$ npx host-consensus 'http://example.com\@evil.com'
http://example.com\@evil.com
  whatwg   example.com
  rfc3986  evil.com
  verdict  host-differs
$ echo $?
1

The seven constructions

Over a deterministic enumeration of 312 URLs (2 schemes x 26 authority constructions x 6 path suffixes), 84 rows are read as a different host by the two conventions. Seven distinct authority constructions sit behind them, and each one works differently:

| authority | WHATWG reads | RFC 3986 reads | why | |---|---|---|---| | example.com\@evil.com | example.com | evil.com | backslash is a slash in a special scheme | | 0x7f.1 | 127.0.0.1 | 0x7f.1 | hexadecimal IPv4 canonicalisation | | 017700000001 | 127.0.0.1 | 017700000001 | octal IPv4 canonicalisation | | 2130706433 | 127.0.0.1 | 2130706433 | decimal IPv4 canonicalisation | | ①.com | 1.com | ①.com | UTS-46 mapping | | exаmple.com (Cyrillic а) | xn--exmple-4nf.com | exаmple.com | IDNA to ASCII | | example。com (U+3002) | example.com | example。com | ideographic full stop separates labels |

The IPv4 rows run the other way from the backslash row: http://2130706433 is 127.0.0.1 to a private-range filter written against new URL(), and an opaque hostname to anything RFC-derived. Whichever side does the blocking, the other side disagrees about what it is blocking.

A further 48 rows across four constructions are an acceptance difference, where one convention refuses the URL outright and the other returns a host: exa mple.com, example.com:65536, example.com:+80, example.com: 80.

The remaining 180 rows are reported as agreeing. That matters as much as the rest: differences in spelling, such as IPv6 brackets and host case, are normalised away and never reported, so ordinary URLs come back clean.

What this does not do

It does not resolve DNS, classify addresses as private, or make requests. Those are a different job, and request-filtering-agent does them well, blocking requests to private ranges at the agent level. Use it for that; use this to check that the URL string you validated is the one your next hop will read.

The two are complementary: a private-range filter answers "is this address safe", and host-consensus answers "will the next parser agree this is even the same address".

API

function inspect(url: string): Consensus;
function agrees(url: string): boolean;
function assertConsensus(url: string): string;   // the agreed host, or throws
class HostDisagreementError extends Error { readonly consensus: Consensus }

verdict is one of:

• 'agreed': every convention names the same host.
• 'host-differs': all parsed, and they name different hosts.
• 'acceptance-differs': at least one refused and at least one accepted.

inspect throws TypeError on a non-string and nothing else. agrees never throws on a malformed URL: a predicate that throws gets read as false by the next person to use it, and "this URL is dangerous" must not be reachable by accident. assertConsensus also throws when both conventions refuse, because there is no host to return.

inspect performs no I/O at all (no network, no filesystem, no clock) and is a pure function of its argument.

How the RFC 3986 reading is checked

It is not checked against anybody's reading of the specification. src/corpus.ts records the host and port that CPython 3.11.15's urllib.parse.urlsplit returned for all 312 URLs, and the test suite asserts this package matches real CPython on every one of them. See VERIFY.md to reproduce the headline claim from a clean install in under a minute.

Also from this author

yaml-drift reports every value whose meaning changed converting YAML to JSON; host-consensus reports hosts that two URL conventions read differently. Same shape, a boundary that quietly changes meaning, in a different place.

License

MIT