TypeScript Job Scraper 📝

ts-jobspy is a job scraping library for JavaScript/TypeScript that aggregates jobs from popular job boards with one call.

It began as a TypeScript port of python-jobspy. As of v3 it has diverged into its own project with its own API. The last upstream-parity release line is preserved on the python-jobspy-parity branch (v2.x), and v1 lives on v1-legacy. Migrating from v2? See MIGRATION.md.

Features

• Scrapes Indeed & LinkedIn concurrently (more boards ship as they become reliably scrapable - see Site status)
• Honest results: every scrape reports per-site status, counts, timing, and errors in result.meta - a blocked or failing site can never silently vanish
• Strict input validation: invalid options throw a descriptive InvalidInputError instead of silently coercing to defaults
• Cross-site deduplication (opt-in): by URL or by normalized title + company + location
• Failure isolation: one site erroring never discards another site's jobs (opt into all-or-nothing with strict: true)
• Proxy rotation support (HTTP/HTTPS/SOCKS)
• Fully typed - the options and result schemas are plain, documented TypeScript types

Installation

npm install ts-jobspy

Node.js >= 20.18.1 required (the floor set by the cheerio/undici dependency chain)

Usage

import { scrapeJobs } from 'ts-jobspy';
import fs from 'node:fs';

const result = await scrapeJobs({
  sites: ['indeed', 'linkedin'], // default: the currently working sites
  searchTerm: 'software engineer',
  location: 'San Francisco, CA',
  resultsWanted: 20,
  hoursOld: 72,
  country: 'usa',
  dedupe: 'content', // drop the same posting syndicated across boards
  // linkedin: { fetchDescription: true }, // richer LinkedIn data (slower)
});

console.log(`Found ${result.jobs.length} jobs`);
for (const site of result.meta.sites) {
  console.log(`${site.site}: ${site.status}, ${site.jobs} jobs in ${site.durationMs}ms`);
}
fs.writeFileSync('jobs.json', JSON.stringify(result.jobs, null, 2));

Example Output

| site | company | title | location | datePosted | jobUrl | interval | minAmount | maxAmount | currency | isRemote | |----------|---------|---------------------------------------------------------|-------------------|------------|--------------------------------------------|----------|-----------|-----------|----------|----------| | indeed | Adobe | Software Development Engineer | San Jose, CA, US | 2026-01-02 | https://www.indeed.com/viewjob?jk=17cf2... | yearly | 139000 | 257550 | USD | false | | linkedin | Google | Software Engineer, Infrastructure, User Personalization | Mountain View, CA | 2025-12-31 | https://www.linkedin.com/jobs/view/4326... | yearly | 141000 | 202000 | USD | false |

Options

All options are optional. Invalid values throw InvalidInputError up front - nothing is silently ignored.

scrapeJobs(options)
├── sites (SiteName | SiteName[]):
│    'indeed' | 'linkedin' | 'ziprecruiter' | 'glassdoor' | 'google' | 'bayt' | 'naukri' | 'bdjobs'
│    default: the currently working sites (indeed, linkedin)
├── searchTerm (string): at most 1000 characters
├── location (string): at most 500 characters
├── distance (number): search radius in miles, default 50
├── jobType (string): fulltime, parttime, internship, contract, ...
├── isRemote (boolean)
├── easyApply (boolean): jobs hosted on the board itself
├── resultsWanted (number): per site, default 15, max 10000
├── offset (number): skip this many results per site, max 100000
├── hoursOld (number): only jobs posted in the last N hours
├── country (string): Indeed/Glassdoor country, default 'usa'
├── descriptionFormat ('markdown' | 'html' | 'plain'): default 'markdown'; 'html' is the
│     board's markup with script/style/iframe elements, event handlers and script URLs
│     removed - still treat it as untrusted content when rendering
├── enforceAnnualSalary (boolean): convert hourly/monthly wages to annual
├── dedupe ('none' | 'url' | 'content' | boolean): default 'none'
│    'url' = exact URL match; 'content' (= true) = normalized title+company+location
├── strict (boolean): reject the whole call if any requested site fails or is interrupted; default false
├── timeoutMs (number): abort a site's scrape after this many ms and report it as an error;
│    default none, max 2147483647 (the largest delay a Node timer accepts)
├── siteConcurrency (number): how many sites are scraped in flight at once;
│    default all requested sites concurrently, 1 = sequential (gentler on your IP)
├── proxies (string | string[]): 'user:pass@host:port' (http), 'https://...', 'socks5://...'
│    or 'socks4://...'; rotated per request; each entry must form a usable proxy URL;
│    at most 1000 entries of at most 2048 characters
├── caCert (string): path to a PEM certificate bundle, trusted for every HTTPS connection
│    (direct, through an http(s) proxy, or through a SOCKS proxy); validated when the call starts
├── userAgent (string): at most 1024 characters, no control characters
├── verbose (0 | 1 | 2): 0 errors only (default), 1 +warnings, 2 +info
├── linkedin ({ fetchDescription?, companyIds? }): LinkedIn-specific options; at most 100 companyIds
└── google ({ searchTerm? }): verbatim Google Jobs query, at most 1000 characters

Result schema

scrapeJobs() resolves to a ScrapeResult:

ScrapeResult
├── jobs: Job[]                  // sorted by site, then newest first
│   ├── id, site, jobUrl, jobUrlDirect
│   ├── title, company, location, datePosted (YYYY-MM-DD)
│   ├── jobTypes: string[]       // real arrays, not comma-joined strings
│   ├── salarySource ('direct_data' | 'description'), interval, minAmount, maxAmount, currency
│   ├── isRemote, jobLevel, jobFunction, listingType
│   ├── emails: string[], description
│   ├── companyIndustry, companyUrl, companyLogo, bannerPhotoUrl, companyUrlDirect,
│   │   companyAddresses, companyNumEmployees, companyRevenue, companyDescription
│   └── skills: string[], experienceRange, companyRating,   // Naukri-specific
│       companyReviewsCount, vacancyCount, workFromHomeType
└── meta
    ├── sites[]: { site, status, jobs, requested, durationMs, jobsPerSecond,
    │              error?, unsupportedOptions? }
    │     status: 'ok'      - jobs returned, no interruptions
    │             'empty'   - a valid results page with zero jobs (no matches); a bot
    │                         wall, a redirect off the board, or an unrecognized
    │                         payload is reported as 'error', never as 'empty'
    │             'partial' - some jobs collected, then interrupted (error says why)
    │             'error'   - failed before collecting anything (error says why)
    │     unsupportedOptions: options you set that this site cannot honor
    │              (e.g. Bayt ignores jobType) - present only when non-empty, so a
    │              dropped filter is never silent. See the option support matrix below.
    ├── totalDurationMs
    ├── jobsPerSecond            // overall throughput (each site also reports its own)
    ├── failureRate              // failed/interrupted sites / requested sites, 0..1
    └── duplicatesRemoved

Every field a site provides is passed through - filtering is yours to do.

Concurrency model & throughput

Node.js runs a single thread with asynchronous I/O - there is no multithreading or multiprocessing here, and none is needed: scraping is network-bound, so overlapping requests is what matters. The siteConcurrency option picks the strategy: by default all requested sites are scraped concurrently; 1 scrapes them one at a time (slower, but gentler on your IP against rate limits).

Every scrape reports its own metrics in meta - jobsPerSecond per site and overall, durationMs, and failureRate - so throughput is measurable on every call, not just in benchmarks. node scripts/benchmark.mjs (in the repo) runs the strategy comparison live; from a residential IP (2026-08-31, 15 jobs/site):

| Strategy | Indeed | LinkedIn | Overall | Failure rate | |----------|--------|----------|---------|--------------| | concurrent (default) | ~1067 jobs/min | ~157 jobs/min | ~313 jobs/min | 0 | | sequential (siteConcurrency: 1) | ~577 jobs/min | ~115 jobs/min | ~193 jobs/min | 0 |

LinkedIn's rate is bounded by its own guest-API politeness delays; Indeed's GraphQL API returns 100 jobs per request and dominates throughput.

Reliability, stealth & security notes

• Rate limiting: a 429 or block surfaces as status: 'error'/'partial' with a RateLimitException message in meta.sites[].error - it never silently looks like an empty result. HTTP status codes are not auto-retried (only transport errors are), so a block is reported rather than amplified into a burst.
• Pacing: LinkedIn requests - including per-job description fetches - are jittered. Use siteConcurrency: 1 and modest resultsWanted to stay under rate limits; add proxies for large scrapes.
• Fingerprint: requests send browser-like headers but Node's TLS stack, which sophisticated anti-bot systems can still fingerprint. For heavy or sensitive scraping, route through residential proxies. userAgent customizes LinkedIn/HTML scrapers; Indeed's GraphQL API requires its fixed app user-agent, so it is left untouched there.
• Input safety: searchTerm/location are safely escaped into Indeed's GraphQL query (no injection). Options are strictly validated before any request, and unknown option keys are rejected rather than silently ignored.
• caCert: path to a regular PEM file, read synchronously when the options are validated (every certificate block must parse, else InvalidInputError) and again when each site's session is created. It is trusted for every TLS connection: a direct HTTPS request, the hop to an https:// proxy, and the connection tunneled through an http(s):// or SOCKS proxy. The proxy agents only apply TLS options to the proxy hop by themselves, so the CA is injected into the tunneled connection explicitly.
• Response limits: a response body is capped at 16 MB after decompression (a small gzip body that inflates to hundreds of MB is rejected instead of exhausting the process heap), a redirect that leaves the board's domain is refused with an error naming the destination instead of being followed, and a page or description nested deeper than 256 elements (as the HTML parser would build it: <div/> opens an element, stray end tags are ignored, optional end tags are implied) or longer than 2 million characters is refused (page) or reduced to its text (description; HTML-escaped when the caller asked for html) rather than handed to the HTML parsers, whose cost grows quadratically with nesting depth. Every request also has a total deadline of twice its inactivity timeout (the Indeed API and LinkedIn search pages use a 10 s inactivity timeout, so 20 s; LinkedIn job pages 10 s; other requests 60 s), because a server that trickles a byte every few seconds would otherwise hold a scrape open indefinitely even without timeoutMs. Failures that cannot succeed on retry (body over the cap, refused redirect, deadline, cancelled request) are not retried.
• timeoutMs: threads an AbortSignal into every scraper's requests and paced delays, so a timeout actually aborts in-flight work, not just the wait. If a site had already collected some jobs when the timeout fired, they are returned as status: 'partial' rather than discarded.

Site status

Live status is verified daily by a scheduled scrape-health workflow that alerts when a board changes behavior.

| Site | Status | Notes | |------|--------|-------| | Indeed | ✅ Working | GraphQL API; fastest scraper (~30 jobs/sec), minimal rate limiting | | LinkedIn | ✅ Working | Guest API; rate limits around the 10th page - use proxies for large scrapes | | Google | 🚧 Blocked | Serves a JS-required interstitial to non-browser clients (the jobs data itself is unchanged); the scraper detects the wall and reports error, never a misleading empty | | Glassdoor | 🚧 Blocked | TLS fingerprinting; may work with residential proxies | | ZipRecruiter | 🚧 Blocked | TLS fingerprinting; US/CA only | | Bayt | 🚧 Blocked | TLS fingerprinting | | Naukri | ⚠️ Untested | India-focused; worked at last verification | | BDJobs | 🚧 Moved | The site migrated to an Angular SPA at bdjobs.com/h/jobs (backed by apiv1.bdjobs.com); the old search URL redirects there, which the scraper detects and reports as error until it is ported |

Blocked/untested sites can still be requested - the per-site meta entry will tell you exactly what happened (error with the reason, or empty only when the board genuinely answered with no matches).

Per-site option support

searchTerm, offset, resultsWanted, and descriptionFormat are honored by every site (Bayt and BDJobs are keyword-only and ignore location). The filter options below vary by board; when you set one a site cannot express, it is listed in that site's meta.sites[].unsupportedOptions (never dropped silently):

| Site | location | distance | jobType | isRemote | easyApply | hoursOld | |------|:--------:|:--------:|:-------:|:--------:|:---------:|:--------:| | Indeed | ✅ | ✅ | ✅ (fulltime, parttime, contract, internship) | ✅ | ✅ | ✅ | | LinkedIn | ✅ | ✅ | ✅ (fulltime, parttime, contract, temporary, internship) | ✅ | ✅ | ✅ | | Google | ✅ | ❌ | ✅ * | ✅ * | ❌ | ✅ (coarse) * | | Glassdoor | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ (day) | | ZipRecruiter | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (day) | | Naukri | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ (day) | | Bayt | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | BDJobs | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |

* Google: setting google.searchTerm replaces the assembled query, so jobType, isRemote, and hoursOld are then listed in that site's unsupportedOptions.

Notes: Indeed's API accepts only one filter group per search (hoursOld, or easyApply, or jobType/isRemote together); when you combine them the lower-precedence ones are reported in unsupportedOptions. "(day)" means the site filters at whole-day granularity, so a sub-day hoursOld is applied as one day. "(coarse)" means Google maps hoursOld to broad buckets (today/3 days/week/month). A jobType value a board cannot express (for example perdiem, nights, summer, volunteer) runs the search unfiltered and is reported in unsupportedOptions. Support for the blocked sites reflects what their request-building code sends and is not verified live.

Limitations

├── Indeed: only ONE of these filter groups per search:
│    - hoursOld
│    - jobType & isRemote
│    - easyApply
├── LinkedIn: rate limits at ~10th page; `isRemote` filter is applied
│    inconsistently by LinkedIn's own guest API
└── All boards cap a given search at ~1000 jobs

Supported countries (Indeed)

LinkedIn searches globally and uses only location. Indeed uses country:

Argentina, Australia, Austria, Bahrain, Bangladesh, Belgium, Brazil, Bulgaria, Canada, Chile, China, Colombia, Costa Rica, Croatia, Cyprus, Czech Republic, Denmark, Ecuador, Egypt, Estonia, Finland, France, Germany, Greece, Hong Kong, Hungary, India, Indonesia, Ireland, Israel, Italy, Japan, Kuwait, Latvia, Lithuania, Luxembourg, Malaysia, Malta, Mexico, Morocco, Netherlands, New Zealand, Nigeria, Norway, Oman, Pakistan, Panama, Peru, Philippines, Poland, Portugal, Qatar, Romania, Saudi Arabia, Singapore, Slovakia, Slovenia, South Africa, South Korea, Spain, Sweden, Switzerland, Taiwan, Thailand, Turkey, Ukraine, United Arab Emirates, United Kingdom (uk), United States (usa), Uruguay, Venezuela, Vietnam (the CountryName type exported from the package is the source of truth)

FAQ

Q: Why is Indeed returning unrelated roles? A: Indeed searches descriptions too. Use -word to exclude and "exact phrase" to match:

searchTerm: '"engineering intern" software summer (java OR python OR c++) 2026 -tax -marketing'

Q: Getting HTTP 429? A: You're rate limited. Wait between scrapes and/or pass proxies to rotate IPs. With v3, a rate-limited site shows up as status: 'error' in meta.sites instead of failing the whole call.

Q: How do I know if a job board changed and broke scraping? A: Check result.meta.sites - a site that used to return ok and now returns empty/error has changed or blocked you. This repo's daily health workflow watches for the same drift on our side.

Credits

Started as a TypeScript port of python-jobspy by Cullen Watson and Zachary Hampton.

Author: Alpha Romer Coma ([email protected])

License

MIT License - see LICENSE.