simple-url-cache

Conditionally cache your URL's content on REDIS with RegExp. Also supports cache instance sharing and isolation.

Installation

npm install simple-url-cache

API

• CacheEngine
  • methods
    • constructor
    • url()
    • clearDomain()
    • clearInstance()
    • getStoredHostnames()
    • getStoredURLs()
  • static properties
    • helpers
• CacheStorage
  • setters/getters
    • delete()
    • get()
    • has()
    • set()
  • methods
    • getCategory()
    • getDomain()
    • getInstanceName()
    • getStorageType()
    • getUrl()
• Configuration
  • Cache config
  • Redis storage config

Cache Engine

setters /getters

constructor

constructor( defaultDomain: string, instanceName: string, storageConfig: Object, cacheConfig: Object)

defaultDomain Every URL that miss a hostname will get classified under this domain

instanceName The isolated instance where this cacheEngine will store urls. If another cacheEngine has the same storage type and the same instance name, they will share the pool.

storageConfig Redis Storage Config Defines how & where url content is stored

cacheConfig Cache config Supports TTL and inclusion/exclusion for any URL rules you need

Example:

var CacheEngine = require('simple-url-cache');

var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules); 
var engine2 = new CachEngine('http://localhost:4444', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules);
var engine3 = new CachEngine('http://localhost:5555', 'I2', {host: '127.0.0.1',port: 6379}, cacheRules);

// At this stage, engine1 and engine2 share the same pool.

engine1.url('http://a.com/index.html').set('some content'); 
// resolve(true)

engine2.url('http://b.com/index.html').set('some content');
// resolve(true)

engine1.url('http://b.com/index.html').set('some content'); 
// resolves(false) - already cached

engine3.url('http://a.com/index.html').set('some content');
// resolve(true)

engine1.url('http://b.com/index.html').get() 
//resolve(true) -> shared pool with engine1 

engine3.url('http://b.com/index.html').get()
// reject(false) -> not set

url

url(url: string): CacheStorage

url Initialize a new CacheStorage instance ready to be get(), set(), delete() and has().

clearDomain

clearDomain(domain: string): Promise<boolean>

Delete all the cached urls stored within this instance under the specified domain.

clearInstance

clearInstance(): Promise<boolean>

Removes all the cached URLs for all domains for this instance.

getStoredHostnames

getAllCachedURL(): Promise<string[]>

Retrieves an array of all the domains cached.

example:

var CacheEngine = require('simple-url-cache');

var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules); 

engine1.url('http://a.com/index.html').set('content').then( ... )
engine1.url('http://b.com/index.html').set('content').then( ... )

CacheEngine.getStoredHostnames().then(function(results) {
    console.log(results);
    // ['http://a.com', 'http://b.com']
});

domain if none provided, then the default domain will be used

getStoredURLs

getCachedDomains(idomain:string): Promise<string[]>

Get the array of cached URLs associated with this domain & instance

domain All the stored URLs retrived had this domain prepended

example:

var CacheEngine = require('simple-url-cache');

var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules); 

engine1.url('http://a.com/index.html').set('content').then( ... )
engine1.url('http://a.com/about.html').set('content').then( ... )

CacheEngine.getStoredURLs().then(function(results) {
    console.log(results);
    // ['/index.html', '/about.html']
});

Static helper

The methods used to validate the CacheConfig and the RedisStorageConfig objects are exposed statically.

They all throw aTypeError when invalid

validateCacheConfig()

validateCacheConfig(config: CacheRules)

validateRedisStorageConfig()

validateRedisStorageConfig(config: RedisStorageConfig)

CacheStorage

geters & setters

delete

delete(): Promise<boolean>

Resolve to true if the url has been suppressed, false if the url wasn't cached Reject an Error if any

get

get(): Promise<string>

Resolve to the url's content Reject if the url wasn't cached

has

has(): Promise<boolean>

Resolve to true if the url is cached, false if the url is not cached, rejected on error

set

set(content: string [, force: boolean]) : Promise<boolean>

Resolve to true if the url has been cached successfully, Rejects false if - the url matches the never rule. - The url has already been cached Rejects on Error

html: the content of the url to be cached, must be UTF8

force: - Actualize the TTL for maxAge already cached urls - Force the caching for url matching the never rule.

methods

getCategory()

Returns the url's internal category name. always, maxAge or never

getDomain()

Returns the domain which the URL has been stored with.

var url = CacheEngine.url('http://a.com/index.html');
url.set('content').then()

url.getDomain() // http://a.com

getInstanceName()

The instanceName set when this url has been stored

var CacheEngine = require('simple-url-cache');

var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules); 
var engine2 = new CachEngine('http://localhost:3333', 'I2', {host: '127.0.0.1',port: 6379}, cacheRules); 

var url1 = engine1.url('http://a.com/index.html')
var url2 = engine1.url('http://a.com/about.html')

url1.getInstanceName() // I1
url2.getInstanceName() // I2

getStorageType()

Same as getInstanceName(), will return redis

Storage engines

So far, only redis is supported, but it is not hard to add more, PR are welcome.

Initially, FileSystem storage was supported, but it has been removed for several reasons :

• Performances issues.
• Huge complexity issues when dealing with large sets of data, specially when getStoredURLs() is called or if a power outage happens.

it had to replay the whole Regex test against each stored URL, and then make a stat on the file in case it matches a maxAge rule to check the creation time.

But if you need to add another storage engine, like mongo for example, the code is designed in a way were the CacheStorage and CacheEngine APIs are completly storage independent.

Config Files

Cache Config

This is an object describing which URL will be cached, which URLs won't be cached, and which ones will have a ttl expiration.

This is the same object, independently of the storage engine used.

An example worth 1000 words :

exports.cacheConfig = {
    // Will cache all URL starting with /posts/ and ending with html for 24 hours
    cacheMaxAge: [ 
        {
            regex: /^\/posts.*html$/,  
            maxAge: 3600
        }
    ],
    // Will cache about-us.html, contact-us.html and /prices.html indefinitively
    cacheAlways: [  
        {
            regex: /^about-us\.html$/, 
            regex: /^contact-us\.html$/,
            regex: /^prices\.html$/
        }
    ],
    // will never cache the url /sitemaps.html
    cacheNever: [ 
        {
            regex: /^sitemaps\.html$/
        }
    ], 
    // If no URL is matched against these rules, then the default is to never cache it. can be 'never' or 'always'
    default: 'never' 
};

Redis storage config

A bit more complex. The library noderedis is used here, so a valid redis node config file is needed.

example :

export.redisStorageConfig = {
    host: '127.0.0.1',
    port: 6379,
    socket_keepalive: true
}