LinkedIn Industry Sleuth Agent

Explores industry professionals by keyword — skills, titles, or any phrase people actually use on LinkedIn — and pulls the key information for industry exploration.

Point it at a category and it answers four questions: what does this field call itself, who works in it, who monetizes it, and who does both.

The category is configuration, not code. One YAML file is the only thing that changes between industries.

Install

npx linkedin-industry-sleuth init      # scaffold into the current directory
npx linkedin-industry-sleuth doctor    # check prerequisites
npx linkedin-industry-sleuth list      # preview what init writes

Then configure and run:

cp config/industry.example.yaml config/industry.yaml   # edit name + keywords

In Claude Code: /linkedin-industry-sleuth for all phases, or --step N for one.

Artifacts

A run writes four files into data/. They are not committed — the directory ships empty and you generate your own.

[KEYWORD_*] below is replaced by the industry.name in your config. Set it to Solar Finance and phase 2 writes data/Solar Finance Professionals.csv.

| File | What it holds | |---|---| | [KEYWORD_MAP].md | Tiered synonym map for the category, with sources | | [KEYWORD_PROFESSIONALS].csv | LinkedIn profiles positioning in the space | | [KEYWORD_OFFERINGS].csv | Courses, cohorts and certifications, links verified | | [KEYWORD_TAKEAWAYS].md | Synthesized brief: what the data says, and its limits |

Professional columns: first_name, last_name, profile_url, headline, bio, location, company, website_url, thought_leader — headline is the one-liner under someone's name, bio is their About section, empty for profiles that have none. thought_leader is marked x for anyone who also creates an offering in the offerings CSV. Phase 5 adds industry_function, role_type, seniority and region.

The professionals CSV only keeps profiles whose bio carries a tracked keyword. LinkedIn's search matches whole profiles, so roughly half of all search hits do not meet that bar; those land in data/raw/professionals_no_keyword_in_bio.csv alongside the other scrape checkpoints. Nothing excluded is deleted — the rejects are often the more interesting read.

The seven phases

| # | Phase | Cost | |---|---|---| | 1 | Keyword map | Web search only | | 2 | People discovery | LinkedIn quota + account risk | | 3 | Offerings | Web search + fetch | | 4 | Cross-reference | Local only | | 5 | Classification | Local + agent judgment | | 6 | Synthesis | Agent judgment | | 7 | Publishing | Local, optional |

Phases 1, 3, 4, 5 and 6 are cheap and safe to re-run. You should review phase 2 and make sure you're ok with the risks and costs. Everything about its design exists to make interruption cheap.

Manually:

.venv/bin/python scripts/browser.py                              # verify session
.venv/bin/python scripts/discover_people.py --limit 5 --pages 1  # phase 2 dry run
.venv/bin/python scripts/discover_people.py                      # phase 2
.venv/bin/python scripts/discover_offerings.py                   # phase 3
.venv/bin/python scripts/cross_reference.py                      # phase 4
.venv/bin/python scripts/classify.py                             # phase 5

LinkedIn access

Phase 2 drives a real headed Chromium against a persistent profile in .browser-profile/. You log in by hand on first run; nothing stores credentials.

Automated browsing is against LinkedIn's user agreement, so the scraper is deliberately conservative: randomized human-paced delays, a 150-profile-per-run cap, read-only page loads, and a hard stop on any captcha or commercial-use-limit page. Both scraped profiles and discovered URLs are checkpointed as it goes, so an aborted run resumes without re-spending search quota. The browser page is rebuilt every 35 profiles because Chromium's renderer reliably crashes under sustained LinkedIn use.

Every cap and delay lives in config/industry.yaml. Raise them only after a clean run, and only a little. Read .claude/skills/linkedin-industry-sleuth/references/ethics-and-limits.md before pointing this at LinkedIn.

Setup

brew install [email protected]
/opt/homebrew/bin/python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/playwright install chromium

Worked example

config/industry.example.yaml ships filled in with a real mapping of the agentic go-to-market category — the run this tool was extracted from. It's there because a working example is easier to adapt than an empty schema. Replace it with your own category; nothing in the pipeline is specific to that one.

License

Apache 2.0. Not affiliated with LinkedIn Corporation. See NOTICE for the binding terms on automated access and your responsibilities as an operator.