@webupro/medusa-scraper

Google Maps lead scraping for Medusa v2: an admin page to run searches, a poller that ingests results, global dedupe, and CSV/XLSX export.

The plugin talks to a gosom/google-maps-scraper instance over its REST API. The scraper is not part of this plugin — it runs as its own service, and one instance can serve many shops.

Why the scraper runs separately

It used to be a service inside one shop's docker-compose.yml. That coupling caused a real outage: the scraper does not resume in-flight work after a restart, and a redeploy of the shop's backend recreated the whole stack — including the scraper — so every scrape running at that moment was stranded at working forever, reporting 0 results with nothing to explain why.

Running the scraper as its own project fixes that, and lets several shops share one instance.

Install

npm install @webupro/medusa-scraper

Register it in medusa-config.ts:

module.exports = defineConfig({
  plugins: [
    {
      resolve: "@webupro/medusa-scraper",
      options: {},
    },
  ],
})

Then apply the migrations:

npx medusa db:migrate

Configuration

| Env | Required | Default | Meaning | | --- | --- | --- | --- | | SCRAPER_URL | yes in production | http://gmaps-scraper:8080 | Base URL of the scraper's REST API | | SCRAPER_PROXIES | no | — | Comma-separated egress proxies passed to each job |

The default only works when the scraper sits on the same Docker network under that hostname.

Reaching a shared scraper

The scraper's REST API has no authentication. POST /api/v1/jobs is open, and /download returns whatever leads a job collected. Its safety therefore comes entirely from not being reachable — do not put it on a public domain without a gate in front of it.

The recommended topology is one scraper per host, joined to the same Docker network as the shops on that host, with no published ports and no ingress labels. Every shop then reaches it at http://gmaps-scraper:8080 with no credentials to leak and no exposure to manage. Shops on another host get their own instance rather than reaching across the internet.

deploy/docker-compose.yml in this repo is that deployment. Point your platform at it rather than pasting an image name into a UI, for one specific reason: the stable address comes from the compose service name, which is registered as a network alias and survives redeploys. Platforms like Coolify rewrite the container name to include a deployment id, so the container itself answers to an address that moves every time it is deployed — and an image-name deployment has nothing else to answer to. With the compose file, gmaps-scraper resolves permanently and SCRAPER_URL never has to be set at all.

If one shared instance across hosts is genuinely needed, put it behind a private mesh (Tailscale/WireGuard) rather than a public domain, so the "unreachable" property is preserved.

Memory

Country-wide runs fan out over 16 cities with email extraction, which is by far the heaviest workload the scraper takes. Size the container accordingly — a 1 GB limit is where the first out-of-memory restarts appeared, and each restart strands every job in flight, across every shop sharing the instance.

What it does

• Admin page at /app/lead-scraper: start searches, watch job status, browse and export leads.
• Coverage: a single area, or country-wide fan-out over the major cities of Slovakia (sk_all) or Hungary (hu_all). The location is appended to each keyword, because the scraper geo-constrains by what is in the keyword rather than by a separate field.
• Polling every minute (lead-scraper-poll). Results are ingested only when the remote job reports done — while it is running, the found count stays 0 because nothing has been downloaded yet.
• Stall guard: a run that outlives its own max_time plus 5 minutes of grace is marked failed. Nothing else ever ends a stranded job: the scraper answers working indefinitely, and neither side times it out.
• Dedupe across every job by a normalised business key, so re-running a search never duplicates a lead.
• Export to CSV and XLSX.

Data and consent

The plugin stores business contact data (name, address, phone, e-mail, website) harvested from public listings. That is personal data when the business is a sole trader, so:

• each lead keeps its source and scraped_at as provenance;
• do_not_contact is a suppression flag — honour it before any outreach;
• retention and lawful basis for contacting these leads are the consuming shop's responsibility, not this plugin's.

Development

npm install
npm run build      # medusa plugin:build
npm test           # unit tests (pure logic: CSV, normalise, dedupe, status)

The REST contract this plugin depends on — including a captured live response from the pinned scraper image — is recorded in src/modules/lead_scraper/__fixtures__/CONTRACT.md. Re-validate it before bumping the scraper image.