athena-query-result-pager

v0.5.1

Published

8 hours ago

Paginate AWS Athena GetQueryResults calls with AWS SDK v3. This library walks NextToken, parses tabular rows with athena-query-result-parser, and forwards MaxResults and QueryResultType from PagerOptions on every GetQueryResults request.

0High
0Medium
0Low

yicr

athena aws aws-sdk pager query result

Athena Query Result Pager

npm license

Paginate AWS Athena GetQueryResults calls with AWS SDK v3. This library walks NextToken, parses tabular rows with athena-query-result-parser (0.5+), and forwards MaxResults, QueryResultType, and optional ParseResultSetOptions from PagerOptions on every page fetch.

Page-level and row-level APIs are symmetric: use the base method for raw ParsedRow values, or the *With / rowParser variant for custom types.

Compared to the AWS SDK paginator

@aws-sdk/client-athena already ships paginateGetQueryResults. It walks NextToken and yields raw GetQueryResultsCommandOutput pages — fine if you only need SDK-shaped pages and will parse rows yourself.

This package is the easier path when you want rows, not raw Athena pages: header skipping, dictionary / typed rows, and page- or row-level async iteration are built in, so you skip the usual glue code around the official paginator.

| | SDK paginateGetQueryResults | This package | | --- | --- | --- | | Page walking (NextToken) | Yes | Yes | | Output | Raw Athena / SDK page objects | Parsed ParsedRow (or custom T via RowParser) | | Header-row handling & parse options | You implement | Via athena-query-result-parser (skipHeaderRow, mismatch behavior, …) | | Row-level async iteration | You flatten pages | iterateRows (one page buffered at a time) | | Defaults for MaxResults / QueryResultType | Per request input | Constructor PagerOptions, applied every fetch | | Input checks | SDK only | Fail-fast for empty queryExecutionId and invalid maxResults |

Both still require a finished query execution and rely on your AthenaClient for retries — see Caller responsibilities.

Features

Sends paginated Athena GetQueryResults requests (NextToken / MaxResults).
Forwards QueryResultType from the pager constructor on each request — accepts the full AWS SDK enum (QueryResultType.DATA_ROWS, QueryResultType.DATA_MANIFEST, …). Use QueryResultType.DATA_MANIFEST for CTAS / UNLOAD / INSERT manifest workflows when Athena allows it.
Forwards parseResultSetOptions to every parser invocation — columnCountMismatchBehavior, skipHeaderRow, headerRowDetectionStrategy, unavailableResultBehavior, maxRows, and other parser 0.5+ options.
Paired APIs for raw ParsedRow dictionaries versus typed rows via RowParser<T>:
- Page level: fetchPage / fetchPageWith, iteratePages / iteratePagesWith
- Row level: iterateRows (no parser) / iterateRows (with rowParser)
Async generators iterate page-by-page or row-by-row without holding full result sets in memory (only one page of rows is buffered at a time for row iterators).
Fail-fast validation: maxResults must be an integer in 1..1000; queryExecutionId must be non-empty (fetchPage, fetchPageWith).
Re-exports QueryResultType, parser types (ParseResultSetOptions, ParsedRow, RowParser<T>, …), and parser utilities (rowToTypedObject, parseResultSetOnce, EXTRA_COLUMNS_KEY, …) from the package entry point.
Helpers: AthenaQueryResultPager.hasNextPage, getLastHeaderRowDecision(), and reset() (usually optional — parser state resets automatically when queryExecutionId changes).
Does not wait for query completion or add Athena-specific retry beyond the AWS SDK client — see Caller responsibilities.

Installation

npm install athena-query-result-pager

yarn add athena-query-result-pager

pnpm add athena-query-result-pager

Runtime dependencies: @aws-sdk/client-athena, athena-query-result-parser ^0.5.0. Your application typically already instantiates AthenaClient with credentials and Region.

Usage

Create a pager instance

AthenaQueryResultPager normalizes options at construction time (defaults: maxResults 1000, queryResultType QueryResultType.DATA_ROWS). maxResults and queryResultType are sent on every GetQueryResults call; parseResultSetOptions (when set) is applied on every page parse.

import { AthenaClient } from '@aws-sdk/client-athena';
import { AthenaQueryResultPager, QueryResultType } from 'athena-query-result-pager';

const client = new AthenaClient({ region: 'us-east-1' });
const pager = new AthenaQueryResultPager(client, {
  maxResults: 1000,
  queryResultType: QueryResultType.DATA_ROWS,
});

// Manifest-style outputs (only for supported Athena query types):
const manifestPager = new AthenaQueryResultPager(client, {
  queryResultType: QueryResultType.DATA_MANIFEST,
});

// Parser options (athena-query-result-parser 0.5+), applied on every page parse:
const strictPager = new AthenaQueryResultPager(client, {
  parseResultSetOptions: {
    columnCountMismatchBehavior: 'throw',
    headerRowDetectionStrategy: 'safe',
    unavailableResultBehavior: 'throw',
  },
});

Fetch one page (raw rows)

const page = await pager.fetchPage('query-execution-id');
console.log(page.rows, page.nextToken, page.rowCount);

if (AthenaQueryResultPager.hasNextPage(page)) {
  const nextPage = await pager.fetchPage('query-execution-id', page.nextToken);
  // ...
}

Fetch one page with a custom row parser

import type { RowParser } from 'athena-query-result-pager';

interface MyRow {
  id: string;
  name: string;
}
const rowParser: RowParser<MyRow> = (row) => {
  if (row.name == null || row.name === '') return null;
  return { id: row['id'] ?? '', name: row.name };
};

const page = await pager.fetchPageWith('query-execution-id', rowParser);
// page.rows is MyRow[] (rows where rowParser returned null are omitted)

Iterate pages (async generator)

// Raw ParsedRow pages
for await (const page of pager.iteratePages('query-execution-id')) {
  console.log(page.rowCount, page.rows);
}

// With custom row parser
for await (const page of pager.iteratePagesWith('query-execution-id', rowParser)) {
  console.log(page.rowCount, page.rows);
}

Iterate row by row (memory-efficient)

// Raw ParsedRow rows (delegates to iteratePages)
for await (const row of pager.iterateRows('query-execution-id')) {
  console.log(row);
}

// With custom row parser (delegates to iteratePagesWith)
for await (const row of pager.iterateRows('query-execution-id', rowParser)) {
  console.log(row);
}

Inspect header-row skipping

When parseResultSetOptions.skipHeaderRow is 'auto', check whether the first row was skipped:

await pager.fetchPage('query-execution-id');
const decision = pager.getLastHeaderRowDecision();
console.log(decision?.skipped, decision?.reason);

Reusing a pager across executions

Switching to a different queryExecutionId on fetchPage / fetchPageWith (and iterators that call them) automatically resets the bundled parser so header-row handling does not leak between queries. Call reset() only when you need to clear parser state without starting a new fetch:

pager.reset();

Caller responsibilities

This library only paginates and parses GetQueryResults. It does not start queries, wait for execution to finish, or wrap Athena errors with custom retry policies. Failures surface as thrown AWS SDK errors (or parser errors) from fetchPage / fetchPageWith.

Handle the following in the calling application (or via AthenaClient configuration):

Query not finished yet — Before paging, wait until GetQueryExecution reports a terminal state such as SUCCEEDED. Calling this pager while the query is still QUEUED / RUNNING will fail with the usual Athena / SDK error for that situation.
Throttling and transient API errors — Configure retry on the AthenaClient (AWS SDK v3 retry middleware / client maxAttempts, etc.) or retry at the call site around fetchPage / iterators. This package does not implement Athena-specific backoff for ThrottlingException or similar.
Failed / cancelled queries — Check execution state (and Athena’s error reason) before or when catching failures; the pager assumes a completed execution that can return result pages.

Options

`PagerOptions`

maxResults?: number
- Default 1000.
- Valid range: integer 1..1000 (Athena GetQueryResults limit).
- Throws RangeError when invalid (constructor).
queryResultType?: QueryResultType
- Type is the AWS SDK QueryResultType enum (QueryResultType.DATA_ROWS, QueryResultType.DATA_MANIFEST, …) — not restricted to a single literal.
- Import QueryResultType from athena-query-result-pager (re-exported from @aws-sdk/client-athena).
- Default QueryResultType.DATA_ROWS.
- When set (including default), the value appears on fetchPage, fetchPageWith, and iterator-driven calls alongside MaxResults.
parseResultSetOptions?: ParseResultSetOptions
- Forwarded to every AthenaQueryResultParser.parseResultSet / parseResultSetWith call on this pager.
- Use for columnCountMismatchBehavior ('silent' | 'throw' | 'warn' | 'extra'), skipHeaderRow, headerRowDetectionStrategy, duplicateColumnNames, unavailableResultBehavior, maxRows / maxRowsExceededBehavior, and other options documented in athena-query-result-parser.
- Import ParseResultSetOptions from athena-query-result-pager (re-exported).
- Inspect header skipping via pager.getLastHeaderRowDecision() after a fetch.

Method input validation

fetchPage(queryExecutionId, nextToken?)
fetchPageWith(queryExecutionId, rowParser, nextToken?)

Both throw if queryExecutionId is empty or whitespace-only (before invoking AWS).

API