Apify Test Tools

Contributing link

Getting Started

1. Install the package npm i -D apify-test-tools
   • because it uses annotate, vitest version to be at least 3.2.0
   • make sure that target and module in your tsconfig.json's compilerOptions are set to ES2022
2. create test directories: mkdir -p test/platform/core
   • core (hourly) tests should go to test/platform/core
   • daily tests should go to test/platform
3. setup github worklows TODO

File structure:

google-maps
├── actors
└── src
└── test
    ├── unit
    └── platform
        ├── core                  <- Core tests need to be inside core directory
        │   └── core.test.ts
        ├── some.test.ts          <- Other tests can be defined anywhere inside platform directory
        └── some-other.test.ts

Github worklows

There should be 4 GH workflow files in .github/workflows.

platform-tests-core.yaml

name: Platform tests - Core

on:
    schedule:
        # Runs at the start of every hour
        - cron: '0 * * * *'
    workflow_dispatch:

jobs:
    platformTestsCore:
        uses: apify-store/github-actions-source/.github/workflows/platform-tests.yaml@new_master
        with:
            subtest: core
        secrets: inherit

platform-tests-daily.yaml

name: Platform tests - Daily

on:
    schedule:
        # Runs at 00:00 UTC every day
        - cron: '0 0 * * *'
    workflow_dispatch:

jobs:
    platformTestsDaily:
        uses: apify-store/github-actions-source/.github/workflows/platform-tests.yaml@new_master
        secrets: inherit

pr-build-devel-test.yaml

name: PR Test

on:
    pull_request:
        branches: [master]

jobs:
    buildDevelAndTest:
        uses: apify-store/github-actions-source/.github/workflows/pr-build-test.yaml@new_master
        secrets: inherit

release-latest.yaml

name: Release latest

on:
    push:
        branches: [master]

jobs:
    buildLatest:
        uses: apify-store/github-actions-source/.github/workflows/push-build-latest.yaml@new_master
        secrets: inherit

Writing tests

Test structure

testActor runs the actor and provides extended expect and run inside the callback.

import { describe, testActor } from 'apify-test-tools';

describe('test', () => {
    testActor(actorId, 'actor test 1', async ({ expect, run }) => {
        const runResult = await run({ input });

        // your checks
    });

    testActor(actorId, 'actor test 2', async ({ expect, run }) => {
        const runResult = await run({ input });

        // your checks
    });
});

Validating basic run attributes

toFinishWith validates common run properties in a single call:

await expect(runResult).toFinishWith({
    datasetItemCount: 100,
});

You can also specify a range:

await expect(runResult).toFinishWith({
    datasetItemCount: { min: 80, max: 120 },
});

Here is full example of what you can validate with toFinishWith

await expect(runResult).toFinishWith({
    // These are default
    status: 'SUCCEEDED',
    duration: {
        min: 600, // 0.6 sec
        max: 600_000, // 10 min
    },
    failedRequests: 0,
    requestsRetries: { max: 3 },
    forbiddenLogs: ['ReferenceError', 'TypeError'],

    // only datasetItemCount is required
    datasetItemCount: { min: 80, max: 120 },

    // optional
    chargedEventCounts: {
        'actor-start': 1,
        'place-scraped': 9,
    },
});

Custom validations

expect(place.title, `London Eye's title`).toEqual('lastminute.com London Eye');

Custom validation functions

You can create your own functions wrapping a common validation logic in e.g. test/platform/utils.ts and import it in test files.

import { ExpectStatic } from 'apify-test-tools'

export const validateItem = (expect: ExpectStatic, item: any) {
    expect(item.title, 'Item title').toBeString();
}

Test options

You can pass options as the fourth argument to testActor:

testActor(
    actorId,
    'slow actor test',
    async ({ expect, run }) => {
        const runResult = await run({ input });
        await expect(runResult).toFinishWith({ datasetItemCount: 100 });
    },
    {
        timeout: 2 * 60 * 60 * 1000, // 2 hours (default is 1 hour)
        retry: 3, // retry up to 3 times (default is 1)
    },
);

Using prefilled input

If the actor has a prefilled input on the platform, you can merge it with your test input:

testActor(actorId, 'with prefilled input', async ({ expect, run }) => {
    const runResult = await run({
        prefilledInput: true,
        input: { maxItems: 10 }, // merged on top of the prefilled input
    });
    await expect(runResult).toFinishWith({ datasetItemCount: 10 });
});

Testing an existing run

You can skip starting a new run and validate an existing one by passing runId:

testActor(actorId, 'validate existing run', async ({ expect, run }) => {
    const runResult = await run({ runId: 'some-run-id' });
    await expect(runResult).toFinishWith({ datasetItemCount: 100 });
});

Accessing run data

RunTestResult provides methods to access the run's data:

testActor(actorId, 'check dataset items', async ({ expect, run }) => {
    const runResult = await run({ input });

    // Access dataset items
    const { items } = await runResult.getDataset();
    expect(items[0].title).toBeNonEmptyString();

    // Access run log
    const log = await runResult.getLog();
    expect(log).toContain('Crawl finished');

    // Access crawler statistics
    const stats = await runResult.getStatistics();
    expect(stats?.requestsFinished).toBeGreaterThan(0);

    // Access key-value store
    const kvs = runResult.getKeyValueStoreClient();
    const record = await kvs.getRecord('OUTPUT');

    // Access run info (refreshed from API)
    const runInfo = await runResult.getRunInfo();
});

Testing standby actors

Use testStandbyActor for actors that support standby mode:

import { describe, testStandbyActor } from 'apify-test-tools';

describe('standby tests', () => {
    testStandbyActor(actorId, 'standby request', async ({ expect, callStandby }) => {
        const { data, status } = await callStandby({
            input: { query: 'test' },
            path: '/search',
            headers: { 'Content-Type': 'application/json' },
        });

        expect(status).toBe(200);
        expect(data.results).toBeNonEmptyArray();
    });
});

Custom matchers

testActor extends expect with the following custom matchers:

• toBeArray() / toBeEmptyArray() / toBeNonEmptyArray()
• toBeString() / toBeNonEmptyString() / toStartWith(prefix)
• toBeNumber() / toBeWholeNumber() / toBeWithinRange(min, max)
• toBeBoolean() / toBeTrue() / toBeFalse()
• toBeObject() / toBeNonEmptyObject()
• toFinishWith(options) - validates run status, duration, dataset, logs, etc.

CLI (apify-test-tools bin)

The package includes a CLI binary used by CI workflows to build Actors, detect changes, and report test results. You can also run it locally.

Running locally

Running the testing library locally is useful when you only want to update the testing code in /test because you can iterate on it without pushing new code to the remote.

If you don't need to change any source files and only iterate on /test code, you can skip steps 1-4. But if you want to test vs changed /src, you have to push that GitHub branch since it needs to build the Actors with that code.

The main local flow is:

1. Switch to a dummy branch that you will push and can later delete
2. npm i apify-test-tools@latest -D
3. Push your code (changes you want to test)
4. Build Actors on Apify (with your new code)
5. Run tests against those builds. You can change tests and run on the same builds.

cd into the actor repository you want to work with (or use --workspace).

4. Build affected Actors

If you want to test vs existing src code, you can skip this and instead construct the output JSON manually from existing builds only for the Actors you need to test.

Requires APIFY_TOKEN_<USERNAME> for all Apify users that own your Actors (e.g. apify, compass, lukaskrivka users). The username is derived from the actor name — uppercased with non-word chars replaced by _ (e.g. Actor john.doe/my-actor needs APIFY_TOKEN_JOHN_DOE).

APIFY_TOKEN_JOHN_DOE=<token> \
GITHUB_WORKSPACE=. \
  npx apify-test-tools build \
    --target-branch origin/master \
    --source-branch origin/my-dummy-branch \
    --dry-run

Remove --dry-run to actually trigger builds and update the branch names/ The command outputs a JSON array of build objects to stdout:

[{ "buildId": "...", "actorId": "...", "buildNumber": "...", "actorName": "john.doe/my-actor" }]

5. Run tests against the builds

Pass the build output as ACTOR_BUILDS and provide TESTER_APIFY_TOKEN. The token can point to your own account (if you have enough memory) or you can use the testing account (xRGg9iAfJSymqartk).

If you want to run only certain tests, change the test/platform to be more specific.

ACTOR_BUILDS='<JSON output from build command>' \
TESTER_APIFY_TOKEN=<token> \
RUN_PLATFORM_TESTS=1 \
  npx vitest --run --maxConcurrency 20 --fileParallelism=true --maxWorkers 100 test/platform

Full example

# Build and capture output
BUILDS=$(APIFY_TOKEN_JOHN_DOE=apify_api_xxx \
  GITHUB_WORKSPACE=. \
  npx apify-test-tools build \
    --target-branch origin/master \
    --source-branch origin/my-dummy-branch)

# Run tests with the builds
ACTOR_BUILDS="$BUILDS" \
TESTER_APIFY_TOKEN=apify_api_yyy \
RUN_PLATFORM_TESTS=1 \
  npx vitest --run --maxConcurrency 20 --fileParallelism=true --maxWorkers 100 test/platform

Dev mode

For development on apify-test-tools itself, use tsx directly:

GITHUB_WORKSPACE=local-clone tsx bin/main.ts get-actor-configs