raspar
v3.0.2
Published
A simple to use Promise-based web scraper with local caching.
Downloads
23
Readme
raspar
A simple to use Promise-based web scraper with local caching.
Usage
const { fetch } = require('raspar');
fetch('http://www.google.com/humans.txt').then(content => console.log(content));
const { fetch } = require('raspar');
fetch([
'http://www.google.com/humans.txt',
'http://www.google.com/robots.txt'
]).then(content => console.log(content[0]));
Options
const { fetch } = require('raspar');
const options = {
cacheDirectory: 'temp/cache/',
requestOptions: {
headers: {
'User-Agent': 'request'
},
method: 'POST'
},
ttl: 1800
};
fetch('http://www.google.com/humans.txt', options).then(content =>
console.log(content)
);
| Name | Description | Default Value |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| cacheDirectory | Directory to store cache. | temp/cache/
|
| requestOptions | Request options object. Read more github.com/node-fetch/node-fetch | {}
|
| ttl | TTL (Time to live) in seconds. | 1800
|