datadirect-puppeteer

An implementation of scraping Blackbaud's front-end APIs using Puppeteer

Install

npm install @msar/datadirect-puppeteer datadirect puppeteer

Realistically, you'll want access to the datadirect types and will need to configure puppeteer, so they are peer dependencies.

Usage

import { api as types } from 'datadirect';
import { api, PuppeteerSession } from '@msar/datadirect-puppeteer';

const session = await PuppeteerSession.Authenticated.getInstance(
  'https://example.myschoolapp.com',
  {
    username: '[email protected]',
    password: 's00p3rS3kre7'
  }
);

// optional to explicitly type `groups`, as the `datadirect-puppeteer` method maps types correctly!
const groups: types.datadirect.groupFinderByYear.Response =
  await api.datadirect.groupFinderByYear({
    session,
    payload: {
      schoolYearLabel: '2024 - 2025'
    }
  });

const topics = await api.datadirect.sectiontopicsget({
  session,
  payload: {
    format: 'json',
    active: true,
    future: false,
    expired: false,
    sharedTopics: true
  },
  pathParams: { Id: 12345678 }
});

Headless Impersonation

If you are using a PuppeteerSession.Impersonation, I recommend running your script with Puppeteer in headless mode. Empirically, this seems to make it less likely that stray user activity will interfere with the Puppeteer-controlled UI actions necessary to make the impersonation work in the browser.

import { api, PuppeteerSession } from '@msar/datadirect-puppeteer';
const session = await PuppeteerSession.Impersonation.getInstance(
  'https://example.myschoolapp.com',
  {
    username: '[email protected]',
    password: 's00p3rS3kre7',
    headless: true,
    val: '12345678',
    searchIn: 'UserID'
  }
);
const conversations = await api.message.inbox({
  session,
  payload: {
    format: 'json',
    pageNumber: 1,
    toDate: '01/20/2025'
  }
});

Known Issues

Actual error-generating issues are tagged bug, all others are improvements or documentation issues.