athena-query-result-pager

v0.4.4

Published

4 days ago

Paginate AWS Athena GetQueryResults calls with AWS SDK v3. This library walks NextToken, parses tabular rows with athena-query-result-parser, and forwards MaxResults and QueryResultType from PagerOptions on every GetQueryResults request.

0High
0Medium
0Low

yicr

athena aws aws-sdk pager query result

Athena Query Result Pager

npm license

Paginate AWS Athena GetQueryResults calls with AWS SDK v3. This library walks NextToken, parses tabular rows with athena-query-result-parser (0.4+), and forwards MaxResults, QueryResultType, and optional ParseResultSetOptions from PagerOptions on every page fetch.

Page-level and row-level APIs are symmetric: use the base method for raw ParsedRow values, or the *With / rowParser variant for custom types.

Features

Sends paginated Athena GetQueryResults requests (NextToken / MaxResults).
Forwards QueryResultType from the pager constructor on each request — accepts the full AWS SDK enum (QueryResultType.DATA_ROWS, QueryResultType.DATA_MANIFEST, …). Use QueryResultType.DATA_MANIFEST for CTAS / UNLOAD / INSERT manifest workflows when Athena allows it.
Forwards parseResultSetOptions to every parser invocation — columnCountMismatchBehavior, skipHeaderRow, headerRowDetectionStrategy, and other parser 0.4+ options.
Paired APIs for raw ParsedRow dictionaries versus typed rows via RowParser<T>:
- Page level: fetchPage / fetchPageWith, iteratePages / iteratePagesWith
- Row level: iterateRows (no parser) / iterateRows (with rowParser)
Async generators iterate page-by-page or row-by-row without holding full result sets in memory (only one page of rows is buffered at a time for row iterators).
Fail-fast validation: maxResults must be an integer in 1..1000; queryExecutionId must be non-empty (fetchPage, fetchPageWith).
Re-exports QueryResultType, parser types (ParseResultSetOptions, ParsedRow, RowParser<T>, …), and parser utilities (rowToTypedObject, EXTRA_COLUMNS_KEY, …) from the package entry point.
Helpers: AthenaQueryResultPager.hasNextPage, getLastHeaderRowDecision(), reset() when reusing one pager across different executions.

Installation

npm install athena-query-result-pager

yarn add athena-query-result-pager

pnpm add athena-query-result-pager

Runtime dependencies: @aws-sdk/client-athena, athena-query-result-parser ^0.4.0. Your application typically already instantiates AthenaClient with credentials and Region.

Usage

Create a pager instance

AthenaQueryResultPager normalizes options at construction time (defaults: maxResults 1000, queryResultType QueryResultType.DATA_ROWS). maxResults and queryResultType are sent on every GetQueryResults call; parseResultSetOptions (when set) is applied on every page parse.

import { AthenaClient } from '@aws-sdk/client-athena';
import { AthenaQueryResultPager, QueryResultType } from 'athena-query-result-pager';

const client = new AthenaClient({ region: 'us-east-1' });
const pager = new AthenaQueryResultPager(client, {
  maxResults: 1000,
  queryResultType: QueryResultType.DATA_ROWS,
});

// Manifest-style outputs (only for supported Athena query types):
const manifestPager = new AthenaQueryResultPager(client, {
  queryResultType: QueryResultType.DATA_MANIFEST,
});

// Parser options (athena-query-result-parser 0.4+), applied on every page parse:
const strictPager = new AthenaQueryResultPager(client, {
  parseResultSetOptions: {
    columnCountMismatchBehavior: 'throw',
    headerRowDetectionStrategy: 'safe',
  },
});

Fetch one page (raw rows)

const page = await pager.fetchPage('query-execution-id');
console.log(page.rows, page.nextToken, page.rowCount);

if (AthenaQueryResultPager.hasNextPage(page)) {
  const nextPage = await pager.fetchPage('query-execution-id', page.nextToken);
  // ...
}

Fetch one page with a custom row parser

import type { RowParser } from 'athena-query-result-pager';

interface MyRow {
  id: string;
  name: string;
}
const rowParser: RowParser<MyRow> = (row) => {
  if (row.name == null || row.name === '') return null;
  return { id: row['id'] ?? '', name: row.name };
};

const page = await pager.fetchPageWith('query-execution-id', rowParser);
// page.rows is MyRow[] (rows where rowParser returned null are omitted)

Iterate pages (async generator)

// Raw ParsedRow pages
for await (const page of pager.iteratePages('query-execution-id')) {
  console.log(page.rowCount, page.rows);
}

// With custom row parser
for await (const page of pager.iteratePagesWith('query-execution-id', rowParser)) {
  console.log(page.rowCount, page.rows);
}

Iterate row by row (memory-efficient)

// Raw ParsedRow rows (delegates to iteratePages)
for await (const row of pager.iterateRows('query-execution-id')) {
  console.log(row);
}

// With custom row parser (delegates to iteratePagesWith)
for await (const row of pager.iterateRows('query-execution-id', rowParser)) {
  console.log(row);
}

Inspect header-row skipping

When parseResultSetOptions.skipHeaderRow is 'auto', check whether the first row was skipped:

await pager.fetchPage('query-execution-id');
const decision = pager.getLastHeaderRowDecision();
console.log(decision?.skipped, decision?.reason);

Reset before another execution

Reuse the same pager for a new queryExecutionId after clearing the bundled parser state:

pager.reset();

Options

`PagerOptions`

maxResults?: number
- Default 1000.
- Valid range: integer 1..1000 (Athena GetQueryResults limit).
- Throws RangeError when invalid (constructor).
queryResultType?: QueryResultType
- Type is the AWS SDK QueryResultType enum (QueryResultType.DATA_ROWS, QueryResultType.DATA_MANIFEST, …) — not restricted to a single literal.
- Import QueryResultType from athena-query-result-pager (re-exported from @aws-sdk/client-athena).
- Default QueryResultType.DATA_ROWS.
- When set (including default), the value appears on fetchPage, fetchPageWith, and iterator-driven calls alongside MaxResults.
parseResultSetOptions?: ParseResultSetOptions
- Forwarded to every AthenaQueryResultParser.parseResultSet / parseResultSetWith call on this pager.
- Use for columnCountMismatchBehavior ('silent' | 'throw' | 'warn' | 'extra'), skipHeaderRow, headerRowDetectionStrategy, duplicateColumnNames, and other options documented in athena-query-result-parser.
- Import ParseResultSetOptions from athena-query-result-pager (re-exported).
- Inspect header skipping via pager.getLastHeaderRowDecision() after a fetch.

Method input validation

fetchPage(queryExecutionId, nextToken?)
fetchPageWith(queryExecutionId, rowParser, nextToken?)

Both throw if queryExecutionId is empty or whitespace-only (before invoking AWS).

API