@sprigr/apps-file-indexing
v0.2.10
Published
The shared file-indexing loop for marketplace apps that mirror a cloud drive into the platform's ACL-enforced file index (`<companyId>-app-<slug>-acl-files`). microsoft-365 (OneDrive and SharePoint), google-workspace (Google Drive) and dropbox use it. The
Readme
@sprigr/apps-file-indexing
The shared file-indexing loop for marketplace apps that mirror a cloud drive into the platform's ACL-enforced file index (<companyId>-app-<slug>-acl-files). microsoft-365 (OneDrive and SharePoint), google-workspace (Google Drive) and dropbox use it. The design and the alternatives it replaced are in sprigr-team decision 0151 (docs/decisions/proposed/0151-cloud-storage-apps-share-one-file-indexing-package-behind-a-file-source-adapter.md).
npm install @sprigr/apps-file-indexing # exact-pin it, like every @sprigr/apps-* package@sprigr/apps-app-sdk is a peer dependency (>=0.14.0 <1), so the package runs on the app's own SDK copy and the app's bundle carries one SDK, not two. Keep @sprigr/apps-app-sdk (0.14.0 or later) in the app's own dependencies, which every app already does. 0.1.0 pinned the SDK at exactly 0.14.0 as a regular dependency, so an app on 0.15 shipped both copies.
An app supplies two things:
- a
FileSourceAdapter: the provider calls (list changes, walk the tree, read permissions, download bytes); - a
FileIndexingStore: durable state.createD1FileIndexingStorebuilds one over the app's own D1 tables, with the table and column names as config, so existing immutable migrations keep working.
The package owns everything that must behave the same for every provider:
| Concern | What the package guarantees |
|---|---|
| Principal grammar | user: / group: / org: / public, byte-identical to the platform (__tests__/doc-acl-platform-parity.test.ts, a fixture generated from sprigr-team packages/shared/src/utils/doc-acl.ts). A principal may hold no whitespace, U+0085 included, which the engine trims and JS does not. keepValidPrincipals(candidates, scope) drops invalid ones and warns once without the values |
| Fail closed | A file whose permissions cannot be read is skipped, never stamped public. A row whose principals do not validate is never sent, so one bad row cannot reject a whole batch |
| Cursor discipline | The cursor moves only after its rows are imported, seen IDs recorded, deletions applied and events emitted. Any failure keeps the old cursor |
| Unreadable permissions | A transient permission failure holds the cursor at that page so the file is retried (sprigr-apps#2419), bounded by MAX_UNRESOLVED_HOLD_MS |
| Content | Text-like files up to 256 KB, native exports, and PDF/OOXML through the platform extract bridge (5 per pass, staged under random single-use keys); pptx drains as a durable job. Binaries of 16 MiB or more stay metadata-only. Text is capped at 32000 chars, and a cut is logged and marked |
| Content backlog | A file the pass could not fetch text for (the deadline passed, the 5-binary cap was spent, its drive answered 429, or the fetch threw) is queued for a content fill and filled by later passes of the same scope, 4 rows at a time (sprigr-apps#2725); outcome.contentPending counts what still waits (sprigr-apps#2702) |
| Reconcile | A completed full walk deletes index rows the walk did not see (diff of data.listIds, never on uncertainty). A completed walk that saw NOTHING (the user emptied the drive) deletes rows only on the second consecutive such walk, each also confirmed by the adapter's confirmEmpty when it has one; see "Empty walks" below (sprigr-apps#2690). An errored, cut or held walk, or one that established no cursor, deletes nothing |
| Events | <prefix>.file.created/updated/deleted on incremental passes, with no per-run cap (sprigr-apps#2521). Pages are emitted in order, but the events WITHIN a page go out with bounded concurrency (DEFAULT_FETCH_CONCURRENCY, 6), so their order is not guaranteed; a subscriber that cares orders by modifiedAt |
| Disconnect | purgeActor switches indexing off, unlinks the owner identity and deletes the scope's rows (sprigr-apps#2355) |
| Sharing-only changes | Opt-in refreshAclPrincipals re-stamps acl_principals with data.partialUpdate({ withAcl: true }), leaving content alone (sprigr-apps#2211) |
| Budgets | Every loop stops STARTING work at its deadline, so every cut lands on a resume point under the 110 s dispatch wall |
A tick
import {
createD1FileIndexingStore,
createDeadline,
drainPendingExtractions,
drainPendingPurges,
fileActorStillConnected,
indexActorFiles,
INDEX_FILES_BUDGET_MS,
} from '@sprigr/apps-file-indexing';
export async function runIndexFiles(env: DropboxEnv) {
const store = createD1FileIndexingStore(env.DB); // the package's default tables
const deadline = createDeadline(INDEX_FILES_BUDGET_MS); // once per tick
await drainPendingPurges(store, env, { deadline }); // unfinished disconnects first
for (const { scope } of await store.listEnabled()) { // stale-first
if (Date.now() >= deadline.at) break; // the rest sort first next tick
if (!(await fileActorStillConnected(dropboxAdapter, scope, env))) continue;
const outcome = await indexActorFiles(dropboxAdapter, store, env, scope, { deadline });
// audit outcome.indexed / skipped / unresolved / error as the app does today;
// outcome.contentPending says how many files still wait for their text
}
await drainPendingExtractions(store, env); // its own 15 s slice
}Timing a tick that runs long
When a tick runs past its budget, the step that was running tells you little.
newCallTiming records each call's count, total and slowest ms, and the ms it
ran past the deadline (late, the column to read):
const timing = newCallTiming({ deadlineAt: deadline.at, now });
const timedEnv = withD1Timing(withCallTiming(env, timing), timing); // files.*, data.*, emit, d1.*
const adapter = timeAdapter(myAdapter, timing, 'ms'); // ms.fullWalk, ms.downloadBinary, ...
await timing.time('step.pass', () => indexActorFiles(adapter, store, timedEnv, scope, { deadline }));
log({ calls: callsForRow(timing.calls), open: timing.open(), untimed: timing.untimed() });callsForRow keeps the slowest calls within a size cap (default 1,800 chars) and counts the rest in _omitted.
Worked example: a Dropbox adapter
Dropbox's listing is cursor-based like Graph delta: files/list_folder (recursive) enumerates everything and ends with a cursor; files/list_folder/continue returns changes since a cursor; a 409 with reset means the cursor is dead and the app must start over. Principals come from sharing/list_file_members, and every file in a shared folder has the same members, so they are cached per shared folder.
import {
groupPrincipal,
userPrincipal,
type ChangePage,
type FileSourceAdapter,
type ResolvedPrincipals,
} from '@sprigr/apps-file-indexing';
type Entry = DropboxMetadata; // { '.tag': 'file' | 'folder' | 'deleted', id, name, path_display, ... }
// A deleted entry carries only a path (no id), so rows are keyed on the
// lowercased path: a deletion then names its own row, and a move arrives as a
// delete of the old path plus an entry at the new one.
const objectId = (key: string, pathLower: string) => `dbx:file:${key}:${pathLower}`;
function toPage(res: ListFolderResult, key: string): ChangePage<Entry> {
const live = res.entries.filter((e) => e['.tag'] !== 'deleted');
const removed = res.entries
.filter((e) => e['.tag'] === 'deleted')
.map((e) => ({ objectID: objectId(key, e.path_lower), path: e.path_display }));
// has_more true: `cursor` continues THIS listing. false: it is the new baseline.
return { entries: live, removed, cursor: res.cursor, hasMore: res.has_more };
}
export const dropboxAdapter: FileSourceAdapter<Entry, DropboxEnv> = {
logLabel: '[dropbox-file-indexing]',
eventPrefix: 'dropbox',
eventIdField: 'fileId',
objectIdsActorScoped: true, // the actor key is in every objectID
cursorResetDetail: 'list_folder_cursor_reset',
objectIdPrefix: (ctx) => `dbx:file:${ctx.key}:`,
async fullWalk(cursor, ctx) {
const res = cursor
? await dbx(ctx, 'files/list_folder/continue', { cursor })
: await dbx(ctx, 'files/list_folder', { path: '', recursive: true, limit: 500 });
if (res.status === 409 && res.error?.['.tag'] === 'reset') return { entries: [], cursor: null, hasMore: false, reset: true };
return toPage(res, ctx.key);
},
async listChanges(cursor, ctx) {
return this.fullWalk(cursor, ctx); // same endpoint; the package knows which pass it is
},
async resolvePrincipals(entries, ctx) {
const cache = new Map<string, string[] | 'unresolved'>();
const out = new Map<string, ResolvedPrincipals>();
for (const e of entries) {
const folder = e.sharing_info?.parent_shared_folder_id ?? e.sharing_info?.shared_folder_id;
if (!folder) {
out.set(objectId(ctx.key, e.path_lower), []); // private: the package adds the owner
continue;
}
if (!cache.has(folder)) {
try {
const m = await dbx(ctx, 'sharing/list_folder_members', { shared_folder_id: folder });
cache.set(folder, [
...m.users.map((u) => userPrincipal(u.user.email)),
...m.groups.map((g) => groupPrincipal(g.group.group_id)),
]);
} catch (err) {
// 429 / 5xx: retry next pass (the cursor holds). A 403 would be 'denied'.
cache.set(folder, isRetryable(err) ? 'unresolved' : []);
}
}
out.set(objectId(ctx.key, e.path_lower), cache.get(folder)!);
}
return out; // shared LINKS stamp nothing: a link confers no search visibility
},
objectIdOf: (e, ctx) => objectId(ctx.key, e.path_lower),
toObject: (e, acl_principals, ctx) => ({
objectID: objectId(ctx.key, e.path_lower),
acl_principals,
name: e.name,
path: e.path_display,
isFolder: e['.tag'] === 'folder' ? 'true' : 'false',
mimeType: e['.tag'] === 'folder' ? 'folder' : mimeFromName(e.name),
size: e.size ?? 0,
fileId: e.id,
source: 'dropbox',
modifiedAt: e.server_modified ?? '',
createdAt: e.server_modified ?? '',
content: '',
}),
downloadText: async (o, ctx) => (await dbxDownload(ctx, String(o.fileId))).text(),
downloadBinary: (o, ctx) => dbxDownload(ctx, String(o.fileId)),
// Asked only after a completed full walk saw nothing while rows remain:
// a fresh, NON-recursive root listing, independent of the walk's cursor.
async confirmEmpty(ctx) {
const res = await dbx(ctx, 'files/list_folder', { path: '', recursive: false, limit: 1 });
if (!res.ok) throw new Error(`confirm_empty: list_folder ${res.status}`); // a failure is never "empty"
return res.entries.length === 0 && res.has_more === false;
},
};Empty walks: when an emptied account sheds its rows
A completed full walk that sees zero entries would make every indexed row look stale, and that is exactly what a broken or transient empty listing looks like too. So an empty walk deletes rows only when ALL of these hold (sprigr-apps#2690):
- The walk completed normally: it reached its final page with no error, deadline cut or permission hold, established a cursor, and the walk state did not move under it (the same gates every reconcile has).
- Two consecutive completed empty walks (
EMPTY_WALKS_BEFORE_PURGE). The first records a marker and sets the cursor to NULL, so the very next pass is another full walk (outcome.emptyWalkFollowUp: true); nobody has to toggle indexing. The second deletes. Any completed full walk that sees entries clears the marker, and so dopurgeActorandstore.enable. - When the adapter has
confirmEmpty, each of those walks is also confirmed independently.confirmEmpty(ctx)is asked at reconcile time and must resolvetrue. It should be a separate, cheap read that proves emptiness (dropbox: a non-recursive rootfiles/list_folderansweringentries: []andhas_more: false).falsedeletes nothing and leaves the marker where it was. A throw deletes nothing, leaves the marker where it was, and recordsreconcile_failed: ...with the cursor kept, so the next pass walks again. Without the hook (microsoft-365 and google-workspace today) the two walks alone decide; adding one makes a broken empty listing unable to wipe an account on its own.
The marker needs no migration: it is one row per counted empty walk in the walk-seen table every app already has, under emptyWalkMarkerKey(walkKey) (<walkKey>#empty-walk), a key no real walk uses, so clearWalkSeen(walkKey) leaves it alone. A custom store needs nothing new either.
The purge lists each prefix (reconcilePrefixes([], ctx) when the adapter has it, else objectIdPrefix(ctx); install-scoped ids with another actor indexing are still skipped), logs a console.warn naming the prefix and row count, then deletes PURGE_DELETE_CHUNK ids at a time while at least MIN_PURGE_LEG_MS of the pass deadline is left. A deadline cut or a truncated listing asks for another full walk, which must again be a completed empty walk (and confirmed, with the hook) before it deletes the rest. With nothing indexed under the prefixes there is nothing to protect: confirmEmpty is not asked and no follow-up walk runs, so a brand-new empty account costs one walk.
Serialising passes: runExclusive
An adapter that must not run two passes over one scope at once (a webhook and the schedule racing on the same cursor) implements runExclusive(scope, walkKey, fn, run). Return { busy: true } (plus purgePending: true when a queued purge is the reason) to skip the pass, or { busy: false, value: await fn() }. The fourth argument (0.1.1) is { env, deadline, now, purpose }: the app env, so a lease kept in D1 reads run.env.DB without a per-env adapter; the tick deadline, to size the lease TTL; and purpose, 'index' from indexActorFiles or 'acl_refresh' from refreshAclPrincipals. An adapter written for 0.1.0 declares three parameters and keeps working.
async runExclusive(scope, walkKey, fn, { env, deadline }) {
if (!(await acquireLease(env.DB, walkKey, deadline))) return { busy: true };
try {
return { busy: false, value: await fn() };
} finally {
await releaseLease(env.DB, walkKey);
}
},Team folders (Dropbox Business) slot in as extraScopes: one ExtraScope per team folder, each re-listed whole per pass, the same way microsoft-365 walks SharePoint libraries.
Content fills: the backlog a pass leaves
A pass fetches text one file at a time and stops STARTING fetches at its deadline, after MAX_EXTRACTIONS_PER_RUN binaries, and for a drive that answered 429. The rows are still imported and the cursor still moves (the cursor discipline above is unchanged); each file left without the text it could have had is recorded in the store's pending-extraction table under the token content-fill:<walkKey>, so no migration is needed. A fetch that throws is recorded too. 0.1.1 imported those files metadata-only and forgot them, so a 300-file burst into Dropbox left 215 files searchable by name only, with the status still ok (sprigr-apps#2702).
Every later indexActorFiles pass of the same scope ends with a drain: after the cursor moves, it reads up to MAX_CONTENT_FILLS_PER_PASS (200) of the scope's rows, least recently touched first, and works on CONTENT_FILL_CONCURRENCY (4) of them at once. A row starts only while at least MIN_ITEM_BUDGET_MS is left of BOTH the pass deadline and the drain's slice; rows in flight when either runs out finish. It shares the pass's extraction cap and 429 set, and it never runs a pass past its deadline. The filled rows go to the index in data.import calls of up to CONTENT_FILL_IMPORT_CHUNK (100) rows, and their queue rows are deleted in one batch per call.
How much of a pass the drain gets (0.1.3, sprigr-apps#2725):
- A pass whose walk fetched no text (nothing changed, or only metadata) and that has a deadline gets
IDLE_CONTENT_FILL_BUDGET_MS(45 s), still capped by the deadline. With no deadline the slice stays 15 s. - A busy pass gets
CONTENT_FILL_BUDGET_MS(15 s). When the scope already had rows waiting at the start of the pass, the walk stops STARTING its own content fetches that long before the deadline (at most half of what is left), so a big walk no longer leaves the drain nothing. The files the walk did not start are queued like any other deferral and the drain fetches them concurrently. contentFillBudgetMs: 0turns the drain off for one pass (the rows wait), and with it the early stop: the walk keeps the whole deadline, exactly as in 0.1.2. Webhook passes use this.
Knobs per call: contentFillBudgetMs, idleContentFillBudgetMs, contentFillConcurrency, maxContentFills.
A 429 stops new rows on that throttle key for the rest of the pass, from a download ({ throttled: true }) or from refetchEntry (an error with status === 429, or whatever adapter.isThrottleError says). Such a row waits without spending an attempt. Measured on virtual time with staging's per-row cost (1.1 s: re-read, permissions, download), 300 queued rows fill in two 60 s scheduled ticks (164, then 136) against 13 per tick in 0.1.2 (__tests__/content-fill-throughput.test.ts).
When the env has env.SPRIGR.log, each drain writes one file_indexing.content_fill row (filled, converted, dropped, waiting, not started, pending before and after, slice, peak in flight, elapsed) to the platform's system logs, so the rate can be read without D1.
Per row the drain:
- re-reads the file when the adapter implements
refetchEntry(object, ctx)(return the provider's current entry, ornullwhen it is gone). A vanished file's row is dropped; the row is re-stamped from the current entry through the same fail-closed stamping as the walk, so its text is written under the principals the file has NOW, with its new revision, name and path. Unreadable permissions leave the row queued. WithoutrefetchEntryit uses the stored record, which every walk that imports the file refreshes and every permission re-stamp updates; - fetches the text the way the walk does (
downloadText,exportNative, or the extract bridge; a pptx turns into a durable extraction job); - imports the row through the validated
withAclimport and deletes the queue row. AwithAclpartial update cannot carrycontent(the platform only allowsacl_principalsthere), so the row is imported whole.
A row is dropped (and logged) when the file vanished (refetchEntry returned null, or downloadText returned { missing: true }), no longer qualifies for text, lost every principal, or its fetch failed MAX_CONTENT_FILL_ATTEMPTS (5) times. A walk that imports the file WITH its text drops the row; a deleted file, the reconcile and purgeActor drop it as well. A file that already has a platform extraction job or an app's own queue row is never overwritten by a fill.
Show the backlog. outcome.contentPending (and countPendingContentFills(store, scope) for a status tool) is the number of files of that scope still searchable by name only; describeContentPending(n) gives the sentence for the agent reading the status ("12 files are searchable by name only until their text is processed; a search by what a file says can miss it until then, so read the file itself before saying its text does not exist."). An app that reports "ok" while this is above zero tells agents the text does not exist.
The store
A new app uses the package's default tables: copy DEFAULT_FILE_INDEXING_SCHEMA_SQL into its first file-indexing migration (the app owns its migrations, which are immutable once published) and call createD1FileIndexingStore(env.DB) with no config. The two existing apps keep their tables:
import { createD1FileIndexingStore, GOOGLE_WORKSPACE_STORE_CONFIG, MICROSOFT_365_STORE_CONFIG } from '@sprigr/apps-file-indexing';
// google-workspace: gw_* tables, page_token, walk_list_token / walk_start_token
const gwStore = createD1FileIndexingStore(env.DB, { ...GOOGLE_WORKSPACE_STORE_CONFIG, redact: redactSecrets });
// microsoft-365: ms_* tables, delta_link, connection_id, acl_refresh_* columns
const msStore = createD1FileIndexingStore(env.DB, {
...MICROSOFT_365_STORE_CONFIG,
walkKey: (s) => fileWalkKey(s.actor, s.connectionId!), // keep in-flight seen sets
redact: redactSecrets,
});| Config | Meaning |
|---|---|
| tables | the indexing, walk-seen and pending-extraction tables |
| cursorColumn | the provider cursor column (page_token, delta_link) |
| connectionColumn | per-connection scope (connection_id); scopes must then carry connectionId |
| walkResumeColumns | a full walk keeps its listing continuation and pre-walk baseline in their own columns (google-workspace 0012). Without them, the walk's continuation is the cursor itself (microsoft-365, dropbox) |
| heldSinceColumn | an INTEGER column that bounds the #2419 cursor hold. Neither app has it yet: add ALTER TABLE <t> ADD COLUMN unresolved_held_since INTEGER |
| aclRefreshColumns | where refreshAclPrincipals keeps its continuation and last completion |
| identityLinkedColumn | an INTEGER column recording the last accepted owner identity link (sprigr-app-kit#227). With it a pass skips acl.linkIdentity for IDENTITY_LINK_BACKOFF_MS (24 h) after a success; without it every pass on a fresh isolate pays the call. No app has it by default: add ALTER TABLE <t> ADD COLUMN identity_linked_at INTEGER in the app's own migration, then pass the name. Changing connected_email, disabling, or a failed or refused link clears it |
Disconnect purge: the package queues the rest
purgeActor(adapter, store, env, scope, { deadline }) switches indexing off first, drops the owner identity link, then deletes the scope's rows from the ACL index until the deadline leaves less than MIN_PURGE_LEG_MS. When it cannot finish (a large index, a truncated listing, a delete error) it returns complete: false with the rows it did remove in removed. Since 0.2.4 it also records each unfinished prefix in the store's pending table (no migration) and names them in queued; call drainPendingPurges(store, env, { deadline }) on every indexing tick, before the walk, and the rest goes on later ticks without the person running the disconnect again (sprigr-app-kit#96). A queued prefix re-checks purgeActor's guards when it drains: if the person reconnected, or another actor now indexes an install-wide prefix, the queued purge is dropped and their rows stay. queued is empty when the purge finished, was skipped (purgeSkipped), or the store has no queue (a custom store without listPendingContentFills); in the last case, queue the prefixes yourself as microsoft-365 does with its ms_file_index_purge table. Because indexing is switched off before any delete, an unfinished purge never re-indexes; it only leaves rows behind until the drain.
drainPendingPurges never throws. It returns { purged, removed, dropped, remaining, errors }, looks at up to MAX_PURGES_PER_DRAIN (20) queued prefixes, starts none once the deadline leaves under MIN_PURGE_LEG_MS, and moves a prefix it could not finish to the back of the queue.
purgeIndexPrefix(env, store, prefix, { deadline }) is one drain pass over one prefix, and it never throws. It returns { removed, complete, truncated, cut, unavailable?, error? }: truncated means the listing stopped at the platform cap, cut means the deadline stopped the deletes, and error carries a listing, delete or cleanup failure, with removed still counting the chunks deleted before it. complete is true only when none of those happened. A queue drain keeps the row queued on !complete and records error as the row's last error when it is set:
const pass = await purgeIndexPrefix(env, store, row.prefix, { deadline });
if (pass.complete) await dequeue(row.prefix);
else await requeue(row.prefix, { removed: row.removed + pass.removed, lastError: pass.error ?? null });It purges nothing, and says why in purgeSkipped, when the rows cannot be told apart by id: 'shared_prefix' (install-scoped objectIDs such as gw:file:<id> with other actors indexing) or 'no_indexing_row' (install-scoped objectIDs and this actor has no indexing row, so nothing under the shared prefix is provably theirs).
Failure modes
outcome.error = 'permissions_unresolved: ...',held: true: some files' permissions could not be read; the cursor waits at that page and retries. AfterMAX_UNRESOLVED_HOLD_MS(6 h, needsheldSinceColumn) the pass skips them and reportspermissions_unresolved_released.import_failed/delete_failed/reconcile_failed/walk_seen_record_failed: the cursor did not move; the next pass re-walks.<cursorResetDetail>: the provider rejected the cursor; it is reset and the next pass does a full walk.busy: true:adapter.runExclusivefound another invocation holding the scope; nothing ran.purgeSkipped: 'shared_prefix'/'no_indexing_row': see "Disconnect purge" above. Indexing is still switched off and the identity unlinked.purgeActorreturnscomplete: false: the prefixes inqueuedfinish on laterdrainPendingPurgescalls; an emptyqueuedmeans the rest is yours (see "Disconnect purge" above). A failed pass is inerrorsaspurge <prefix>: <message>.purgeIndexPrefixreturnserror: the listing or a delete failed;removedcounts what went before it. It no longer throws (0.1.0 did).- Reconcile skipped with "walk state moved": another invocation finished or restarted the walk; its reconcile covers it.
content_fill_record_failed: ...: the files this pass could not fetch text for could not be queued (a store write failed). The pass keeps its cursor and imports nothing, so the next pass re-walks them rather than importing them without text and forgetting them.outcome.contentPending > 0: files imported without their text that later passes are filling; see "Content fills" above. Not an error.extract failed for <id> (<format>) [stage=... branch=... attempt=... redownload=...]: a binary extraction failed and the file keeps empty content. The bracket says where:stage(download, put or extract), how the bytes were staged (branch=streamwhen the download declared a length, elsebuffered), the put attempt, and the re-download retry (0.2.6, sprigr-team#13096): a streamed put that fails with a transient R2 fault (10001, 10043, 503) downloads the file once more and retries with the fresh stream, because the bridge cannot replay a spent stream.redownloadisnot-run,ok,put-failed,download-failed(withredownload_statusfor a non-ok answer) orskipped-deadline. A buffered put is left to the bridge's own retry.
Upgrading from 0.1.0
0.1.1 is additive except for two behaviour changes (items 1 and 4), so bump the pin and check the purge drain:
purgeIndexPrefixreturns failures instead of throwing. A drain that recorded a failure in itscatchnow gets{ complete: false, error }back and itscatchnever runs. Readpass.errorwhere thecatchused to set the row's last error or count a failure. Without that change the row still stays queued and is retried, but the failure is no longer recorded.truncatedandcutnow say why a pass was not complete.@sprigr/apps-app-sdkis a peer dependency (>=0.14.0 <1). Nothing to change if the app already depends on the SDK; its bundle drops the second copy.runExclusivegets a fourth argument,{ env, deadline, now, purpose }. Optional to use.- A completed full walk of an empty source now reconciles (sprigr-apps#2690). 0.1.0 skipped the reconcile whenever the walk saw zero entries, so an emptied drive kept every row, full text searchable. Now every row under the prefix goes, but only when the walk reached its final page with no error, cut or hold and established a cursor. A truncated listing deletes the listed subset, and the next completed walk deletes more. An adapter with
reconcilePrefixesis called with an empty seen list and decides which prefixes that covers (microsoft-365's returns none, so its SharePoint rows stay out of it). A directreconcileWalkcall must pass{ completedWalk: true }to get this. 0.2.5 tightened this rule; see "Upgrading to 0.2.5". drainPendingExtractionsnever starts a row with 0 ms or less left.minItemMs: 0used to start one at exactly the deadline; it now behaves likeminItemMs: 1, so an app that passed1to get that behaviour can keep it or drop it.
Upgrading from 0.1.1
0.1.2 (sprigr-apps#2702) needs no migration: content fills live in the pending-extraction table every app already has. Bump the pin, then:
- Show
contentPending.indexActorFilesnow returnscontentDeferred,contentFilledandcontentPending. WirecontentPending(orcountPendingContentFills(store, scope)) into the status the app shows, withdescribeContentPending, instead of reporting a bare "ok" while files wait for their text. - A store with its own
listPendingExtractionsSQL must leave content-fill rows out (substr(job_token, 1, length('content-fill:')) != 'content-fill:', or!isContentFillToken(row.job_token)).drainPendingExtractionsskips any it is handed, but a listing that returns them can fill its page with them and starve the real extraction jobs. dropbox'swithoutRivieraRowsis one. - Optional: implement
refetchEntryso a fill re-reads the file's current permissions and revision and drops a vanished file at once. - Optional: return
{ text: '', missing: true }fromdownloadTextwhen the provider says the file is gone, so its fill is dropped instead of retried. - Behaviour changes.
EnrichSummary.deferredDeadlineandskippedThrottlednow count only files that could have had text (0.1.1 counted folders and metadata-only files too), andenrichObjectsWithContentrecords its deferrals unless called withrecordDeferred: false.listPendingExtractionsForalso returnsjob_token.store.enableclears the scope's old fills, and the permission re-stamp writes the new principals into waiting rows' stored records too. A binary of 16 MiB or more was never extracted (the old comment said it drained as a job; it did not) and still stays metadata-only. EXTRACTION_DRAIN_BUDGET_MSis unchanged, and a pass never runs past its deadline: the fill drain only uses time the pass has left.
Upgrading from 0.1.2
0.1.3 (sprigr-apps#2725) needs no migration and no code change. Bump the pin; the content-fill drain gets faster on its own. What changes:
- Up to 4 rows in flight. Each row is still re-read, re-stamped and fetched in order, so about 12 source calls can be in flight from one drain. A 429 stops new rows on its key for the pass, and a 429 from
refetchEntryno longer spends one of the row's attempts. Lower it withcontentFillConcurrencyif a source needs it. - An idle scheduled pass drains for up to 45 s (
IDLE_CONTENT_FILL_BUDGET_MS), still inside its deadline. A pass with no deadline keeps 15 s. - A busy pass with a backlog stops its walk's content fetches 15 s early (at most half of what is left), and the drain gets that time. That pass may queue a few more files for a fill than 0.1.2 did, each one row write plus its delete; it only happens while a backlog exists.
contentFillBudgetMs: 0passes (webhooks) are unchanged. MAX_CONTENT_FILLS_PER_PASSis 200 (was 100), and the drain imports in calls of up to 100 rows.- New:
ContentFillOutcome.peakInFlight,EnrichSummary.attempted, thedeadlineoption ofenrichObjectsWithContent, the optionaladapter.isThrottleError, and the optionalenv.SPRIGR.logmember (FileIndexingLogEntry). An app whose env already declares the app-sdk'sSprigrLogFnfits it.
Upgrading from 0.2.3
0.2.4 (sprigr-app-kit#96) needs no migration. Add drainPendingPurges(store, env, { deadline }) to the indexing tick, before the walk. Without that call the queue only fills: purgeActor records unfinished prefixes either way. An app that already queues its own purges (microsoft-365) can keep its queue; the package's rows for the same prefixes are harmless and, drained, finish the same work. PurgeActorResult gains queued.
Upgrading to 0.2.5
0.2.5 (sprigr-apps#2690) needs no migration and no code change. It tightens the empty-walk rule 0.1.1 introduced, which deleted every row under the prefix on the FIRST completed walk that saw nothing. No consumer loses the empty-walk purge; every consumer gains the two-walk rule; adapters may add confirmEmpty for an independent check.
- Rows go on the second consecutive completed empty walk, never the first. The first records a marker and asks for another full walk on the next pass by setting the cursor to NULL, so an emptied account heals one walk later than before 0.2.5. See "Empty walks" above.
- Optional: implement
confirmEmpty. Each of the two walks must then also be confirmed by it;falseor a throw deletes nothing. - The empty-walk purge is bounded by the pass deadline and resumes on the next full walk. The non-empty diff is unchanged.
- New:
adapter.confirmEmpty,outcome.emptyWalkFollowUp,EMPTY_WALKS_BEFORE_PURGE,emptyWalkMarkerKey. A directreconcileWalk(..., { completedWalk: true })call follows the same rule and keeps returning the number of rows removed.
Upgrading to 0.2.6
0.2.6 (sprigr-team#13096) needs no code change. Bump the pin. A streamed staging put that hits a transient R2 fault now re-downloads the file once and retries (one extra provider download, only on that fault, skipped past the tick deadline), and the extract logs name the stage, branch, attempt and retry outcome (see "Failure modes").
API
Grammar: PUBLIC_PRINCIPAL, ACL_PRINCIPALS_ATTR, userPrincipal, groupPrincipal, orgPrincipal, isValidPrincipal, normalizePrincipal, stampedPrincipalsValid.
Content: MAX_CONTENT_BYTES, MAX_CONTENT_CHARS, MAX_EXTRACT_INLINE_BYTES, MAX_EXTRACTIONS_PER_RUN, EXTRACT_STAGING_PREFIX, CONTENT_TRUNCATION_MARKER, extractFormatForMime, isTextLikeMimeType, capText, buildExtractJobToken, extractBinaryFileContent, enrichObjectsWithContent, contentKindFor, fetchObjectContent.
Content fills: CONTENT_FILL_TOKEN_PREFIX, CONTENT_FILL_BUDGET_MS, IDLE_CONTENT_FILL_BUDGET_MS, CONTENT_FILL_CONCURRENCY, CONTENT_FILL_IMPORT_CHUNK, MAX_CONTENT_FILLS_PER_PASS, MAX_CONTENT_FILL_ATTEMPTS, contentFillToken, isContentFillToken, storeSupportsContentFills, recordContentFills, drainContentFills, countPendingContentFills, describeContentPending, syncPendingRecordPrincipals.
Pending purges: PURGE_PENDING_TOKEN, MAX_PURGES_PER_DRAIN, isPurgePendingToken, storeSupportsPurgeQueue, drainPendingPurges, DrainPendingPurgesResult, QueuedPurge.
Passes: indexActorFiles, reconcileWalk, emptyWalkMarkerKey, EMPTY_WALKS_BEFORE_PURGE, drainPendingExtractions, refreshPendingExtractions, forgetPendingExtractions, purgeActor, purgeIndexPrefix, drainPendingPurges, refreshAclPrincipals, aclRefreshDue, countIndexedItems, importFileObjects, emitFileEvents, actorOfFileRow, fileActorStillConnected.
Budgets: TickBudget, mapWithConcurrency, createDeadline, deadlinePassed, budgetBelow, remainingBudgetMs, INDEX_FILES_BUDGET_MS, EXTRACTION_DRAIN_BUDGET_MS, EMIT_BUDGET_MS, MIN_ITEM_BUDGET_MS. The Deadline type is @sprigr/apps-fetch-budget's, so one deadline bounds both the loop and each fetch.
Store: createD1FileIndexingStore, DEFAULT_FILE_INDEXING_STORE_CONFIG, DEFAULT_FILE_INDEXING_SCHEMA_SQL, GOOGLE_WORKSPACE_STORE_CONFIG, MICROSOFT_365_STORE_CONFIG, defaultRedact.
