ASIMOV Bright Data Module

ASIMOV module for data import powered by the Bright Data web data platform.

✨ Features

• Imports structured data from Airbnb, Amazon, Crunchbase, eBay, Facebook, Google, Indeed, Instagram, LinkedIn, Walmart, X (aka Twitter), Yahoo, and YouTube.
• Collects the raw JSON data via the Bright Data API (requires an API key).
• Constructs a semantic knowledge graph based on the KNOW ontology.
• Supports plain JSON output as well as RDF output in the form of JSON-LD.
• Distributed as a standalone static binary with zero runtime dependencies.

🛠️ Prerequisites

• Rust 1.85+ (2024 edition) if building from source code

⬇️ Installation

Installation from PyPI

pip install -U asimov-brightdata-module

Installation from RubyGems

gem install asimov-brightdata-module

Installation from NPM

npm install -g asimov-brightdata-module

Installation from Source Code

cargo install asimov-brightdata-module

👉 Examples

export BRIGHTDATA_API_KEY="..."

Fetching X Profiles

asimov-brightdata-fetcher https://x.com/bright_init   # JSON
asimov-brightdata-importer https://x.com/bright_init  # JSON-LD

Fetching LinkedIn Profiles

asimov-brightdata-fetcher https://www.linkedin.com/in/orlenchner/
asimov-brightdata-fetcher https://www.linkedin.com/company/bright-data/

Fetching Crunchbase Profiles

asimov-brightdata-fetcher https://www.crunchbase.com/organization/brightdata

Fetching Amazon Products

asimov-brightdata-fetcher https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465094279

⚙ Configuration

Environment Variables

• BRIGHTDATA_API_KEY: (required) the Bright Data API key to use

📚 Reference

Installed Binaries

• asimov-brightdata-cataloger: discovers entities via the Bright Data API (not implemented yet)
• asimov-brightdata-fetcher: collects JSON data from the Bright Data API
• asimov-brightdata-importer: collects and transforms JSON into JSON-LD (not implemented yet)

Supported Datasets

Dataset | URL Prefix | JSON | RDF :------ | :--------- | :--: | :--: Airbnb | https://www.airbnb.com/rooms/ | ✅ | 🚧 Amazon | https://www.amazon.com/ | ✅ | 🚧   | https://www.amazon.com/sp?seller= | ✅ | 🚧 Crunchbase | https://www.crunchbase.com/organization/ | ✅ | 🚧 eBay | https://www.ebay.com/itm/ | ✅ | 🚧 Facebook | https://www.facebook.com/events/ | ✅ | 🚧   | https://www.facebook.com/groups/ | ✅ | 🚧   | https://www.facebook.com/marketplace/item/ | ✅ | 🚧   | https://www.facebook.com/share/p/ | ✅ | 🚧 Google | https://www.google.com/shopping/product/ | ✅ | 🚧 Indeed | https://www.indeed.com/cmp/ | ✅ | 🚧 Instagram | https://www.instagram.com/ | ✅ | 🚧   | https://www.instagram.com/p/ | ✅ | 🚧   | https://www.instagram.com/reel/ | ✅ | 🚧 LinkedIn | https://www.linkedin.com/company/ | ✅ | 🚧   | https://www.linkedin.com/in/ | ✅ | 🚧   | https://www.linkedin.com/jobs/ | ✅ | 🚧   | https://www.linkedin.com/posts/ | ✅ | 🚧   | https://www.linkedin.com/pulse/ | ✅ | 🚧 Walmart | https://www.walmart.com/global/seller/ | ✅ | 🚧   | https://www.walmart.com/ip/ | ✅ | 🚧 X (Twitter) | https://x.com/ | ✅ | ✅ Yahoo | https://finance.yahoo.com/quote/ | ✅ | 🚧 YouTube | https://www.youtube.com/@ | ✅ | 🚧   | https://www.youtube.com/watch?v= | ✅ | 🚧 | | |

👨‍💻 Development

git clone https://github.com/asimov-modules/asimov-brightdata-module.git