API List

A package to handle Janis List APIs

Installation

npm install @janiscommerce/api-list

ℹ️ Starting from version 9.0.0, this package includes export functionality using SQS consumers. See the Export Feature section for details.

:warning: Breaking Changes in v9.0.0

Starting from version 9.0.0, entity exports are now handled through SQS consumers instead of direct API calls. This requires additional configuration in your service's serverless.yml file to set up the necessary consumers and infrastructure.

If you need to implement exports for your entities, please refer to the Export Feature section below and follow the migration guide for detailed setup instructions.

Usage

'use strict';

const { ApiListData } = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	get alwaysCallFormatter() {
		return true;
	}

	get fieldsToSelect() {
		return ['name', 'status'];
	}

	get fieldsToExclude() {
		return ['error'];
	}

	get fixedFields() {
		return ['code'];
	}

	get sortableFields() {
		return [
			'id',
			'status'
		];
	}

	get availableFilters() {
		return [
			'id',
			{
				name: 'quantity',
				valueMapper: Number
			},
			{
				name: 'hasSubProperty',
				internalName: (filterConfiguration, mappedValue, originalValue) => `rootProperty.${originalValue}`,
				valueMapper: () => true
			}
		];
	}

	get searchFilters() {
		return [
			'id',
			'quantity'
		];
	}

	async formatRows(rows) {
		return rows.map(row => ({ ...row, oneMoreField: true }));
	}

};

ApiListData

The following getters and methods can be used to customize and validate your List API. All of them are optional.

get modelName()

Returns model name. It is intent to be used to change the model's name and it will not get the model name from endpoint

get alwaysCallFormatter()

This is used to force calling the formatRows() method even if fields or excludeFields are sent to the API.

get fieldsToSelect()

This is used to determinate which fields should be selected from the DB.

Important: The id field is always returned.

If set as false. The parameter fields will be ignored.

If a field is not found in the document it will be ignored.

get fieldsToExclude()

Since 5.8.0

This is used to determinate witch fields must be excluded from the response.

If set as false. The parameter excludeFields will be ignored.

Important: The id field is always returned.

If a field is not found in the document it will be ignored.

get fixedFields()

Since 5.8.0

This is used to determinate witch fields should always be returned.

If a field is not found in the document it will be ignored.

async formatRows(rows)

You can use this to format your records before they are returned.

For example, mapping DB friendly values to user friendly values, add default values, translation keys, etc.

get sortableFields()

This is used to indicate which fields can be used to sort the list. Any other sort field will return a 400 status code.

'use strict';

const { ApiListData } = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	get sortableFields() {
		return ['foo', 'bar'];
	}
};

• /api/entity?sortBy=foo with a single value.

Will sort the list by foo in direction asc that are the default

• /api/entity?sortBy=foo&sortDirection=desc with a single value.

Will sort the list by foo in direction desc

• /api/entity?sortBy[0]=foo&sortBy[1]=bar with a single value.

Will sort the list by foo and bar in direction asc that are the default

• /api/entity?sortBy[0]=foo&sortBy[1]=bar&sortDirection=desc with a single value.

Will sort the list by foo and bar in direction desc

• /api/entity?sortBy[0]=foo&sortBy[1]=bar&sortDirection[0]=desc&sortDirection[1]=asc with a single value.

Will sort the list by foo in direction desc and bar in direction asc. The sortDirection is indexed with sortBy

• /api/entity?sortBy[0]=foo&sortBy[1]=bar&sortDirection[1]=desc with a single value.

Will sort the list by foo in direction asc because is the default value and bar in direction desc

Use sortable field valueMapper to return sorts for apply in database instead of sortable field name

'use strict';

const { ApiListData } = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	get sortableFields() {
		return [
			{
				name: 'foo',
				valueMapper: () => [['bar0', 'asc'], ['bar1']]
				/*
					The function in valueMapper must be return an array of array with strings.
					The first string is a sort name and the second is a sort direction.
					If not pass a sort direction in array, by default use a direction passed by data for 'foo' or the default sort direction.
					The default sort direction is 'asc'.
				*/
			},
			{
				name: 'bar',
				valueMapper: direction => (
					direction ? [['bar0', direction], ['bar1', direction]]: [['bar0', 'asc'], ['bar1']]
				)
				// You can use the direction passed from the data for 'bar' or the default sort direction to make a logic that comes in the function parameter
			}
		];
	}
};

get availableFilters()

This is used to indicate which fields can be used to filter the list. Any other filter will return a 400 status code. Filters can be customized by passing an object with the following properties:

• name: (string) The name of the filter param. This property is required.
• internalName: (string|function) The name of the field, as defined in the model. This should not be defined in case it's equal to name. If it's a function (since 3.1.0), it must return a string and it will receive the following arguments: (filterConfiguration, mappedValue, originalValue)
• valueMapper: (function) A function to be called on the filter's value. This is optional.

Value mappers

Since 3.1.0

This lib also exports some value mappers (to use as valueMapper) so you don't need to implement them yourself.

:warning: Warning startOfTheDayMapper and endOfTheDayMapper are now deprecated. See migration guide.

'use strict';

const {
	ApiListData,
	FilterMappers: {
		booleanMapper,
		dateMapper,
		startOfTheDayMapper,
		endOfTheDayMapper,
		searchMapper,
		customTypeMapper
	}
} = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	get availableFilters() {
		return [
			{
				name: 'canDoSomething',
				valueMapper: booleanMapper // Maps '0', 'false', '', and false to false. Any other value is mapped to true.
			},
			{
				name: 'someExactDate',
				valueMapper: dateMapper // Maps to a date object
			},
			{
				name: 'dateCreatedDay',
				internalName: 'dateCreatedFrom',
				valueMapper: startOfTheDayMapper // Maps to a date object at the start of the day
			},
			{
				name: 'dateCreatedDay',
				internalName: 'dateCreatedTo',
				valueMapper: endOfTheDayMapper // Maps to a date object at the end of the day
			},
			{
				name: 'name',
				valueMapper: searchMapper // Maps to an object like this: { type: 'search', value }
			},
			{
				name: 'isOlderThan',
				internalName: 'age',
				valueMapper: customTypeMapper('greater') // This returns a mapper like this: value => ({ type: 'greater', value })
			}
		];
	}

};

get searchFilters()

Since 3.3.0

This is used to indicate which fields will be used to mapped multiple filters (OR Type) for the same value, using only search as single filter. If it don't exist or return an empty array and try to use search filter will return 400 status code. Can be combined with other filters.

'use strict';

const {
	ApiListData
} = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	get searchFilters() {
		return ['someField', 'otherField'];
	}
};

• /api/entity?filters[search]=1 with a single value.

Will filter the list for someField: 1 or otherField: 1

• /api/entity?filters[search]=fo with a uncompleted word.

Will filter the list for someField: fo or otherField: fo and will do a partial filter (like using searchMapper).

• /api/entity?filters[search]=foo bar with multiples words separated by white spaces.

Will filter the list for someField: foo or someField: bar or otherField: foo or otherField: bar.

get staticFilters()

Since 3.4.0

This is used to set a filter with a fixed value for all requests.

'use strict';

const {
	ApiListData
} = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	get staticFilters() {
		return {
			someExactDate: new Date('2020-02-27T14:23:44.963Z'),
			clients: this.session.clientCode
		};
	}
};

This will add two filters (someExactDate and clients) to the request filters (if any). The static filters will not be overriden by user-provided filters.

async formatFilters(filters)

Since 4.1.0

This is used to programatically alter the filters. It will be executed after parsing static and dynamic filters. If you return a falsy value it will not override them. Otherwise, the return value will be used as filters.

You can use this method, for example, to build complex filters or to ensure that a Date range filter is always being applied.

Since 9.0.0, you can also set this.noResults = true to skip the get() call to the model and the totals calculation. This is useful when you know in advance that no results will be returned.

'use strict';

const {
	ApiListData
} = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	async formatFilters(filters) {

		if(filters.someDateFilterFrom && filters.someDateFilterFrom < new Date('2020-01-01T00:00:00.000Z')) {

			// This will override the someDateFilterFrom filter
			return {
				...filters,
				someDateFilterFrom: new Date('2020-02-27T14:23:44.963Z')
			};
		}

		// In this case it will not override the filters
	}
};

'use strict';

const {
	ApiListData
} = require('@janiscommerce/api-list');

const ProductModel = require('../models/product');

module.exports = class StockApiListData extends ApiListData {

	get availableFilters() {
		return [
			'skuId',
			'productId'
		];
	}

	async formatFilters(filters) {

		// If productId filter is received, we need to get the skus for that product
		// and then filter by skuId
		if(filters.productId) {

			const productModel = new ProductModel();
			const skus = await productModel.get({ filters: { id: filters.productId } });

			// If no skus found for the product, we can skip the get() call
			if(!skus.length) {
				this.noResults = true;
				return filters;
			}

			// Replace productId filter with skuId filter
			return {
				...filters,
				skuId: skus.map(sku => sku.id),
				productId: undefined
			};
		}

		return filters;
	}
};

get customParameters()

This allows you to set custom query parameters for your API.

Can be customized by passing a string or object with the following properties:

• name: (string) The name of the custom param. This property is required.
• valueMapper: (function) A function to be called on the parameter's value. This is optional.

The customParameters and its values will be in this.data to use them in your API.

'use strict';

const { ApiListData } = require('@janiscommerce/api-list');

module.exports = class MyApi extends ApiListData {

    get customParameters() {
        return [
			'someParam', // default string
			{
				name: 'numericParam',
				valueMapper: Number
			}
		];
	}

	async formatRows(rows) {

		// To access the parameter, the information arrives through `this.data`
		if(this.data.numericParam === 1)
			return rows.map(row => ({ ...row, oneMoreField: true })); // Do something with the additional parameter

		return rows;
	}
};

• This will allow the API to use custom query parameters. Example: https://domain.com/api/my-api-list?numericParam=1

async formatSortables(sortables)

Since 5.4.0

This is used to programatically alter the sortables. It will be executed after parsing static and dynamic sortables. If you return a falsy value it will not override them. Otherwise, the return value will be used as sortables.

You can use this method, for example, to build complex sorting.

Since 9.0.0, you can also set this.noResults = true to skip the get() call to the model and the totals calculation. This is useful when you know in advance that no results will be returned.

'use strict';

const {
	ApiListData
} = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	async formatSortables(sortables) {

		return Object.keys(sortables).reduce((currentSorts, key) => {

			// We can use 'customFilter' as an identifier for build a complex sorting
			if(key === 'customFilter') {
				const customSorts = { someField: 'asc', otherField: 'desc' };

				return { ...currentSorts, ...customSorts };
			}

			return { ...currentSorts, [key]: sorts[key] };

		}, {});
	}
};

get maxPageSize()

Since 5.5.0

This getter allow to configure a different maximum page-size than default: 100.

'use strict';

const {
	ApiListData
} = require('@janiscommerce/api-list');

module.exports = class MyApiListData extends ApiListData {

	get maxPageSize() {
		return 500;
	}
};

Reducing responses

Since 5.8.0

An Api defined with ApiListData can be reduced using new parameters fields and excludeFields.

:warning: Warning When a response is reduced, it will not call formatRows(), unless the API's alwaysCallFormatter getter returns true

This parameters will be passed to the model for reducing the response on the database-side.

For the following examples we will be using invented products with the information

[{
	"id": 1,
	"code": "t-shirt-red",
	"name": "Shirt Red",
	"price": 200.5,
	"status": "active"
}, {
	"id": 2,
	"code": "t-shirt-blue",
	"name": "Shirt Blue",
	"price": 200.8,
	"status": "active"
}]

When using fields we are telling the database the specific fields we wanna receive in the response.

Important. When using fields, excludeFields will be ignored.

curl --location -g --request GET 'https://my-catalog.com/api/product?fields[0]=code&fields[1]=name'

// expected output: [{ id: 1, code: 't-shirt-red', name: 'Shirt Red' }, { id: 2, code: 't-shirt-blue', name: 'Shirt Blue' }]

When using excludeFields we are telling the database the specific fields we don't wanna receive in the response.

Important. When using fields, excludeFields will be ignored.

curl --location -g --request GET 'https://my-catalog.com/api/product?excludeFields[0]=price'

// expected output: [{ id: 1, code: 't-shirt-red', name: 'Shirt Red', status: 'active' }, { id: 2, code: 't-shirt-blue', name: 'Shirt Blue', status: 'active' }]

Request Headers

An ApiListData accepts request headers to modify default behavior.

|Header|Description|Default| |--|--|--| |x-janis-page|Configure the page of the list to be consulted|1| |x-janis-page-size|The amount of rows to be returned. (max 100)|60|| |x-janis-totals|The package will calculate total using getTotals(). Supports boolean values (true/false) or limit format (max=X) for limited counting. Since 7.0.0.|false|| |x-janis-only-totals|The package will calculate only total (no list items in response) using getTotals(). Supports boolean values (true/false) or limit format (max=X) for limited counting. Since 7.0.0.|false||

Totals Headers Examples

Both x-janis-totals and x-janis-only-totals headers support two formats:

Boolean format (backward compatible):

• x-janis-totals: true - Calculate total without limit (returns data + total)
• x-janis-totals: false - Don't calculate totals
• x-janis-only-totals: true - Calculate total without limit (returns only total)
• x-janis-only-totals: false - Don't calculate totals

Limited counting format:

• x-janis-totals: max=6000 - Calculate total up to 6000 records (returns data + total)
• x-janis-totals: max=100 - Calculate total up to 100 records (returns data + total)
• x-janis-only-totals: max=6000 - Calculate total up to 6000 records (returns only total)
• x-janis-only-totals: max=100 - Calculate total up to 100 records (returns only total)

When using the max=X format, the getTotals() method will be called with a { limit: X } option to enable efficient counting with database-level limits.

ℹ️ The maximum page size can be modified with maxPageSize() getter

Response Headers

An ApiListData will response the following headers.

|Header|Description| |--|--| |x-janis-page|The page used to perform the get() command| |x-janis-page-size|The page size used in the get() command| |x-janis-total|The total of documents according the filters applied. Calculated with getTotals()|

ℹ️ The total calculation can be obtained using request header x-janis-totals or header x-janis-only-totals as true. Using header x-janis-only-totals will prevent using get() command| and no list items will be returned

List APIs with parents

If you have for example, a list API for a sub-entity of one specific record, the parent will be automatically be added as a filter.

Important: The parent entity must be listed as an available filter

For example, the following endpoint: /api/parent-entity/1/sub-entity, will be a list of the sub-entity, and parentEntity: '1' will be set as a filter.

It could be thought as if it's equivalent to the following request: /api/sub-entity?filters[parentEntity]=1

Export Feature

Since 9.0.0

This package now supports exporting entities through SQS consumers instead of direct API calls. The export process is orchestrated by the Batch service, which sends messages to SQS queues that are processed by consumers in your service.

For complete documentation on the export feature, including architecture details and advanced configuration, please refer to the Batch New Export documentation.

Exported Components

The package exports three main components for implementing exports:

• ExportConsumer: Main consumer that processes export requests. It reads messages from SQS, queries data using your ApiList and Model, and uploads the results to S3 in compressed NDJSON format.
• ExportDLQConsumer: Consumer that handles messages from the Dead Letter Queue (DLQ) when exports fail after retries. It forwards error information to an error queue.
• exportServerlessHelperHooks: Serverless Framework hooks that automatically configure the necessary infrastructure (SQS queues, DLQ, SNS subscriptions, Lambda functions, IAM permissions).

Required Configuration

To enable exports in your service, you need to complete the following steps:

Step 1: Add Hooks to serverless.yml

Add the export hooks to your serverless.js configuration:

'use strict';

const { SQSHelper } = require('sls-helper-plugin-janis'); // eslint-disable-line
const { exportServerlessHelperHooks } = require('@janiscommerce/api-list');

module.exports = helper({
	hooks: [
		// service hooks
		...exportServerlessHelperHooks(SQSHelper)
	]
});

This will automatically configure:

• SQS queue for export requests
• Dead Letter Queue (DLQ) for failed exports
• SNS topic subscription (exportRequested from Batch service)
• Lambda functions for both consumers
• Required IAM permissions (including STS AssumeRole for BATCH_EXPORT_ROLE_ARN)

Step 2: Create Consumer Files

Create the consumer files in your service's event-listeners directory:

File: src/sqs-consumer/export/export-consumer.js

'use strict';

const { ExportConsumer } = require('@janiscommerce/api-list');
const { SQSHandler } = require('@janiscommerce/sqs-consumer');

module.exports.handler = event => SQSHandler.handle(ExportConsumer, event);

File: src/sqs-consumer/export/export-dlq-consumer.js

'use strict';

const { ExportDLQConsumer } = require('@janiscommerce/api-list');
const { SQSHandler } = require('@janiscommerce/sqs-consumer');

module.exports.handler = event => SQSHandler.handle(ExportDLQConsumer, event);

Advanced Configuration

You can customize the export behavior by extending the ExportConsumer class and overriding its getters:

get pageSize()

Controls how many records are fetched per page from the database.

• Default: 10000
• Maximum: 25000

'use strict';

const { ExportConsumer } = require('@janiscommerce/api-list');
const { SQSHandler } = require('@janiscommerce/sqs-consumer');

class MyExportConsumer extends ExportConsumer {
	get pageSize() {
		return 5000;
	}
}

module.exports.handler = event => SQSHandler.handle(MyExportConsumer, event);

get rowsPerFile()

Controls how many records are included in each exported file before creating a new file.

• Default: 250000
• Maximum: 250000

'use strict';

const { ExportConsumer } = require('@janiscommerce/api-list');
const { SQSHandler } = require('@janiscommerce/sqs-consumer');

class MyExportConsumer extends ExportConsumer {
	get rowsPerFile() {
		return 100000;
	}
}

module.exports.handler = event => SQSHandler.handle(MyExportConsumer, event);

get pageSizeByEntity()

Allows you to configure different page sizes for different entities. This is useful when some entities have larger documents or different performance characteristics.

'use strict';

const { ExportConsumer } = require('@janiscommerce/api-list');
const { SQSHandler } = require('@janiscommerce/sqs-consumer');

class MyExportConsumer extends ExportConsumer {
	get pageSizeByEntity() {
		return {
			product: 15000,
			order: 8000,
			shipping: 5000
		};
	}
}

module.exports.handler = event => SQSHandler.handle(MyExportConsumer, event);

Export Features

The export system provides the following features:

• Multi-part file generation: Large exports are automatically split into multiple files when they exceed the rowsPerFile limit. Each file is uploaded separately and progress messages are sent for each part.
• Compressed NDJSON format: Files are generated in .ndjson.gz format (newline-delimited JSON compressed with gzip) for efficient storage and transfer.
• formatRows() support: If your ApiListData class implements formatRows() and the export request includes shouldFormatRows: true, the formatting will be applied to each row before export. Note that dependencies do not format the rows.
• Error handling: Automatic retries (maxReceiveCount: 2) and DLQ processing for failed exports. The ExportDLQConsumer handles messages that fail after all retries.
• S3 upload: Files are automatically uploaded to the configured S3 bucket with unique filenames.
• Progress messages: SQS messages are sent to the result queue for each completed part, including metadata such as part number, row count, and file location.

The consumer automatically:

1. Loads the appropriate ApiListData class from api/[parentEntity]/[entity]/list.js
2. Loads the corresponding Model from models/[entity].js (or uses modelName from ApiList if defined)
3. Processes the export using pagination with the configured pageSize
4. Applies formatRows() if requested
5. Uploads files to S3 as they reach the rowsPerFile limit
6. Sends progress messages for each completed part