@manta-eu/upload

A CLI that uploads files to a Manta datastore from outside the app: a script, a scheduled job, or another system's export. It uses the upload flow the chat uses, so the file lands in the datastore exactly as a chat upload would.

CSV, Parquet, Turtle and N-Triples files are then ingested into the datastore's knowledge graph. Any other type is stored but not ingested.

Use

pnpm manta-upload people.csv orders.parquet --datastore your-datastore

The upload targets hiremanta.com production unless --api-url, --keycloak-url and --keycloak-realm name another deployment, as for manta-codegen.

For each file the CLI prints the file's IRI, its media type and its size.

--bundle stores the files without ingesting them, for files that are loaded together as a bundle.

Log in

Login works as it does for manta-codegen: both CLIs register the same commands from @manta-eu/auth. MANTA_CLIENT_SECRET decides the path.

No secret: a person. Log in once with manta-upload login, which takes the same --keycloak-url and --keycloak-realm (both default to hiremanta.com production); status and logout sit next to it. The upload is attributed to that person. The login is stored and renewed silently, and it is the same stored login manta-codegen uses, so logging in with either CLI serves both. See @manta-eu/auth for how the browser login works, where the login is stored and how long it lasts.

A secret: a machine. The CLI asks for a service account token, from the realm's manta-codegen client unless --client-id names another. Add --scope for a narrower token.

export MANTA_CLIENT_SECRET=...   # never a flag: an argument list is public on the machine

A service account only gets a datastore today if it holds the operator realm role. See the manta-codegen README for that trap and for the azp one.

What the CLI does

• Integrity. The CLI hashes the bytes it uploads and aborts if the file changes while it's being read. A part that storage answers with a 5xx, a 429 or a 408, or whose connection drops, is retried for up to three attempts with backoff; a refusal such as a 403 is not.

What the API checks

• Access. The API refuses the upload if the token may not use --datastore. It never falls back to another datastore.
• Content. For the four ingested types, the API reads the start of the file, and for Parquet the end too. It refuses and deletes a file whose bytes contradict its extension: a .parquet without Parquet's magic bytes, or a .csv, .ttl or .nt that isn't UTF-8 text. Re-save a CSV exported as Windows-1252 or ISO-8859-1 as UTF-8 first.
• A check that can't run. If the API cannot read the file back to check it (a storage error, say), it records the file anyway and leaves the verdict to the ingest.
• Size. The API accepts files up to 5 GiB. The ingest skips a file over its own limit, 512 MiB by default.

How ingesting went

The ingest runs in batches, a few minutes after upload. Each version of a file it handles gets a File Ingest (ingest:IngestRun) node in the datastore:

• prov:used points to the file;
• ingest:status is ingested, unparsable, gone or too_large, with prov:endedAtTime;
• on success, ingest:recordsRead, ingest:recordsDropped and prov:generated, the dataset the rows belong to.

A file whose key can't be written as an IRI gets no node. Agents never mints such a key.