arela-uploader

CLI tool to scan, detect, and upload files to the Arela API with multi-tenant support, automatic document detection, and directory watching.

Installation

npm install -g @arela/uploader

Quick Start

The recommended workflow runs 4 phases in sequence:

arela scan                # 1. Discover files and register metadata
arela identify            # 2. Detect document types from PDF content
arela propagate           # 3. Propagate paths to related files
arela push                # 4. Upload files to storage

Commands

arela scan

Scan the filesystem and register file metadata via the API. Supports streaming discovery and multi-level directory partitioning.

arela scan                     # Basic scan
arela scan --api agencia       # Scan using agencia API
arela scan --count-first       # Count files first for progress %
arela scan --no-stream         # Synchronous discovery (no streaming)

| Flag | Description | Default | |------|-------------|---------| | --api <target> | API target (default, agencia, cliente) | default | | --count-first | Count files before scanning for progress tracking | — | | --no-stream | Use synchronous file discovery | — |

arela identify

Detect document types (e.g. pedimento simplificado) from PDF content using pattern matchers. Runs against existing database records.

arela identify                        # Identify documents
arela identify --batch-size 200       # Larger batches
arela identify --show-stats           # Show performance stats

| Flag | Description | Default | |------|-------------|---------| | --api <target> | API target | default | | -b, --batch-size <size> | Files per batch | 100 | | --show-stats | Show performance statistics | — |

arela propagate

Propagate arela_path from identified pedimento records to related files in the same directory.

arela propagate                       # Propagate paths
arela propagate --batch-size 100      # Process 100 pedimentos per batch
arela propagate --show-stats          # Show statistics

| Flag | Description | Default | |------|-------------|---------| | --api <target> | API target | default | | -b, --batch-size <size> | Pedimentos per batch | 50 | | --show-stats | Show performance statistics | — |

arela push

Upload files to Arela storage, filtered by RFC and/or year. Supports cross-tenant mode.

arela push                                          # Upload all files with arela_path
arela push --rfcs RFC1,RFC2 --years 2023,2024       # Filter by RFC and year
arela push --source-api agencia --target-api cliente # Cross-tenant upload
arela push --folder-structure "prefix/path"          # Add storage path prefix
arela push --no-auto-organize                        # Disable auto-organization

| Flag | Description | Default | |------|-------------|---------| | --api <target> | API target | default | | --scan-api <target> | API for reading scan data | default | | --push-api <target> | API for uploading files | — | | --source-api <target> | Source API (cross-tenant) | — | | --target-api <target> | Target API (cross-tenant) | — | | -b, --batch-size <size> | Files to fetch per batch | 100 | | --upload-batch-size <size> | Concurrent uploads | 10 | | --rfcs <rfcs> | Comma-separated RFCs (overrides PUSH_RFCS) | — | | --years <years> | Comma-separated years (overrides PUSH_YEARS) | — | | --folder-structure <path> | Storage path prefix | — | | --no-auto-organize | Disable automatic file organization | — | | --show-stats | Show performance statistics | — |

arela upload

Legacy upload command with multiple modes (stats-only, RFC-based, run-all-phases).

arela upload --batch-size 10                        # Basic upload
arela upload --upload-by-rfc                        # Upload by RFC
arela upload --run-all-phases                       # Run all phases
arela upload --force-supabase --prefix "folder"     # Direct Supabase upload
arela upload --folder-structure "2024/pedimentos"   # Custom folder structure
arela upload --auto-detect-structure                # Auto-detect year/pedimento from paths

| Flag | Description | Default | |------|-------------|---------| | --api <target> | API target | default | | --source-api <target> | Source API (cross-tenant) | — | | --target-api <target> | Target API (cross-tenant) | — | | -b, --batch-size <size> | Files per batch | 10 | | -p, --prefix <prefix> | Prefix for uploaded files | — | | --folder-structure <structure> | Custom folder structure | — | | --client-path <path> | Override client path for metadata | — | | --auto-detect-structure | Auto-detect folder structure from paths | — | | --auto-detect | Enable document type detection | — | | --auto-organize | Enable automatic file organization | — | | --force-supabase | Force direct Supabase upload (skip API) | — | | --skip-processed | Skip already-processed files | — | | --show-stats | Show performance statistics | — | | --upload-by-rfc | Upload based on RFC values from UPLOAD_RFCS | — | | --run-all-phases | Run all processing phases sequentially | — |

arela watch

Monitor directories for file changes and process them automatically.

arela watch -d "/path/to/dir1,/path/to/dir2"       # Watch directories
arela watch --api cliente                           # Watch with specific API
arela watch --source-api agencia --target-api cliente  # Cross-tenant watch
arela watch --strategy individual                   # Upload each file immediately
arela watch --auto-processing --batch-size 10       # Auto pipeline (scan→identify→propagate→push)
arela watch --dry-run                               # Simulate without uploading
arela watch --poll 5000                             # Use polling (for NFS/remote FS)

| Flag | Description | Default | |------|-------------|---------| | --api <target> | API target | default | | --source-api <target> | Source API (cross-tenant) | — | | --target-api <target> | Target API (cross-tenant) | — | | -d, --directories <paths> | Comma-separated directories to watch | — | | -s, --strategy <strategy> | batch, individual, or full-structure | batch | | --debounce <ms> | Debounce delay in milliseconds | 1000 | | -b, --batch-size <size> | Files per batch | 10 | | --poll <ms> | Use polling with interval in ms | — | | --ignore <patterns> | Comma-separated patterns to ignore | — | | --auto-detect | Enable document type detection | — | | --auto-organize | Enable file organization | — | | --auto-processing | Enable automatic 4-step pipeline | — | | --dry-run | Simulate without uploading | — | | --verbose | Verbose logging | — |

Watch strategies:

• batch (default) — Groups files and uploads periodically
• individual — Uploads each file immediately on change
• full-structure — Preserves full directory structure during upload

arela stats

Collect filesystem statistics without uploading (legacy, wraps upload in stats-only mode).

arela stats --batch-size 10

arela detect

Legacy document detection command. Prefer arela identify and arela propagate instead.

arela detect --batch-size 10                  # PDF detection
arela detect --propagate-arela-path           # Path propagation

arela query

Query the database for file status.

arela query --ready-files     # Show files ready for upload

arela config

Display current configuration for all API targets and settings.

arela config

Global Options

| Flag | Description | |------|-------------| | -v, --verbose | Enable verbose logging | | --clear-log | Clear log file before starting | | --version | Display version number | | -h, --help | Display help |

Multi-Tenant API

Three API targets are available: default, agencia, and cliente. Configure each in your .env file.

# Single-tenant
arela scan --api cliente
arela push --api agencia

# Cross-tenant (read from one API, upload to another)
arela push --source-api agencia --target-api cliente

| Phase | --source-api | --target-api | |-------|-----------------|-----------------| | Scan / Identify / Propagate | ✓ | — | | Push / Upload | — | ✓ |

Environment Variables

Copy .env.template to .env and configure:

API Configuration

ARELA_API_URL=https://your-arela-api.example.com
ARELA_API_TOKEN=your-api-token

ARELA_API_AGENCIA_URL=https://agencia-api.example.com
ARELA_API_AGENCIA_TOKEN=your-agencia-token

ARELA_API_CLIENTE_URL=https://cliente-api.example.com
ARELA_API_CLIENTE_TOKEN=your-cliente-token

Supabase (fallback)

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=your-supabase-key
SUPABASE_BUCKET=your-bucket-name

Upload & File Sources

UPLOAD_BASE_PATH=/path/to/files
UPLOAD_SOURCES=folder1|folder2|folder3
UPLOAD_RFCS=RFC1|RFC2|RFC3
UPLOAD_YEARS=2023|2024

Scan Configuration

ARELA_COMPANY_SLUG=your_company          # Company identifier
ARELA_SERVER_ID=nas01                     # Server/NAS identifier
ARELA_BASE_PATH_LABEL=                    # Optional (auto-derived from base path)
SCAN_EXCLUDE_PATTERNS=.DS_Store,Thumbs.db,desktop.ini
SCAN_BATCH_SIZE=2000                      # Records per API call
SCAN_DIRECTORY_LEVEL=0                    # Directory depth for table partitioning

Push Configuration

PUSH_RFCS=                               # Pipe-separated RFC filter
PUSH_YEARS=                              # Pipe-separated year filter
PUSH_BATCH_SIZE=100                      # Fetch batch size
PUSH_UPLOAD_BATCH_SIZE=10                # Concurrent uploads
PUSH_BUCKET=arela                        # Storage bucket
PUSH_FOLDER_STRUCTURE=                   # Prefix for storage paths

Performance Tuning

MAX_API_CONNECTIONS=10                   # Match your API replica count
API_CONNECTION_TIMEOUT=60000             # Timeout in ms
API_MAX_RETRIES=3
API_RETRY_EXPONENTIAL_BACKOFF=true
BATCH_SIZE=100
BATCH_DELAY=0
MAX_CONCURRENT_SOURCES=2

Watch Mode

WATCH_ENABLED=false
WATCH_DIRECTORY_CONFIGS={"../docs/2023":"prefix-2023","../docs/2024":"prefix-2024"}
WATCH_STRATEGY=batch                     # batch|individual|full-structure
WATCH_DEBOUNCE_MS=1000
WATCH_BATCH_SIZE=10
WATCH_USE_POLLING=false
WATCH_POLL_INTERVAL=100
WATCH_STABILITY_THRESHOLD=300
WATCH_IGNORE_PATTERNS=*.tmp,*.bak,*.swp

Document Detection

The tool detects and classifies documents:

• Pedimento Simplificado — Extracts RFC, pedimento number, patente, aduana, and year from PDF content to compose an arela_path (RFC/Year/Patente/Aduana/Pedimento/)
• Supporting Documents — XML, TXT, and JSON customs-related documents

File Sanitization

Filenames are automatically sanitized before upload:

| Original | Sanitized | |----------|-----------| | Facturas Importación.pdf | Facturas-Importacion.pdf | | File{with}brackets.pdf | File-with-brackets.pdf | | Document ^& symbols.pdf | Document-and-symbols.pdf | | file with spaces.pdf | file-with-spaces.pdf |

Handles accented characters, Korean glyphs, special symbols, and email addresses.

Development

npm test                    # Run tests
npm run test:watch          # Run tests in watch mode
npm run test:coverage       # Run tests with coverage
npm run format              # Format code with Prettier

License

ISC