skyscraper
v0.0.8-pre
Published
Scrape the web.
Readme
Skyscraper is an easy to use web scraper / spider for extracting arbitrary data from web pages by using simple CSS selectors. Skyscraper is super good when you want to evaluate a number of pages that all follow the same format.
Try it out for ya self!
$ npm install skyscraper --saveSynopsis
Skyscraper is based on simple schemas (we call them blueprints), somewhat inspired by Mongoose. One blueprint describes one type of page and is linked to one Skyscraper instance. After you have set up the blueprint, you can start scraping pages.
/**
* Create a new `Skyscraper` instance.
*/
var skyscraper = new Skyscraper;
/**
* Set up the blueprint.
*/
skyscraper.blueprint
.property("offer.title")
.selector("#offer h1")
.string()
.default("No title given")
.property("offer.url")
.selector("#offer h1 a", "href")
.string()
.property("offer.price")
.selector("#offer span.offer-price")
.number()
.unsigned()
.float()
.property("offer.description")
.selector("#offer span.offer-price")
.string()
.trim(1000)
.property("offer.items")
.selector("#offer ul.offer-items li")
.array().string()
.property("offer.features")
.selector("#offer ul.offer-features li")
.selector("#offer ul.offer-features li", "class")
.object()
.string()
.boolean(/available/, /unavailable/)
/**
* Scrape a page.
*/
skyscraper.get("http://www.example.com/offer/12345", function(err, model, html) {
// Everything okay?
if(err) throw err;
// Show the scraped data.
console.log(model); // { offer: { title: "No title given", price: 150.3, ... }}
// Show the original HTML code
console.log(html); // "<html>\n\t<head>\n\t\t<title>example.com</title>..."
});Is it ready for production?™
Though we have not reached v1.0.0 yet, we are pretty optimistic about our Sykscraper. Here are the key facts:
Skyscraper is based on cheerio by Matthew Mueller, which is pretty well tested and gets constantly updated. Cheerio is probably the fastest and most forgiving jQuery-esque implementation for Node.js.
We follow the Semantic Versioning rules to spare you the dependency hell.
We heavily use Skyscraper ourself 24/7 for our web-indexing software. If we discover an issue, we want to get it fixed. Instantly.