Upfly

A stream-based, end-to-end file processing and multi-cloud streaming engine for Node.js.

Stream, optimize, and distribute files to Cloudinary, AWS S3, or GCS on-the-fly with zero temporary file overhead.

Website · Quick Start · Cloud Setup · Error Handling · API Reference · GitHub

What is Upfly?

Upfly is a production-grade file processing engine and middleware designed for modern Node.js applications.

Unlike traditional file upload approaches that buffer entire files into memory (causing heap exhaustion) or write unoptimized temp files to disk (cluttering local storage), Upfly processes and streams files on-the-fly as chunked streams.

It handles form parsing, format conversion, aggressive image optimization, multi-cloud uploading, and automatic error recovery inside a single, unified, non-blocking pipeline.

Core Capabilities

• True Stream-Based Architecture: Incoming multipart streams are intercepted and processed in small chunks. Files are optimized and uploaded as they are received, keeping your V8 memory footprint extremely low even under heavy upload concurrency.
• On-the-Fly Optimization: Powered by Sharp, Upfly dynamically resizes, compresses, and converts images into modern formats (like WebP and AVIF) in a non-blocking worker thread pool.
• Intelligent Multi-Cloud Streaming: Stream different fields to different destinations (e.g. avatar images to AWS S3, raw files to disk, thumbnails to Cloudinary) in a single request, using built-in, zero-temp-file cloud adapters.
• Reliable Failovers (safeFile): An automated backup pipeline monitors the streaming flow. If a network socket drops or image optimization fails, Upfly transparently falls back to saving the original file, guaranteeing zero data loss.

Example Integration

const { upflyUpload } = require('upfly');

app.post('/upload',
  upflyUpload({
    fields: {
      avatar: {
        format: 'webp',
        quality: 80,
        cloudStorage: true,
        cloudProvider: 's3',
        cloudConfig: { region: 'us-east-1', bucket: 'my-bucket', ...credentials }
      }
    },
    safeFile: true // Enable fallback backup pipeline
  }),
  (req, res) => {
    // Upfly handles parsing, optimizing, cloud uploading, and fail-safes.
    res.json({ url: req.files.avatar[0].cloudUrl });
  }
);

Highlights

• Stream-based processing — files processed in chunks, never fully loaded into memory
• Image optimization — Sharp-powered conversion to WebP, AVIF, JPEG, PNG, TIFF, GIF, HEIF
• Multi-cloud storage — built-in support for Cloudinary, AWS S3, and Google Cloud Storage
• Zero data loss — automatic backup stream with intelligent fallback when safeFile is enabled
• Flexible output — store to memory, disk, or cloud per field
• TypeScript support — full type definitions with conditional cloud config types
• Multer-compatible — uses Multer as a peer dependency with a custom storage engine under the hood

Install

npm install upfly multer

Requirements: Node.js >= 18 · Multer is a peer dependency — you must install it alongside upfly.

Quick Start

Convert uploaded images to WebP and store them in memory:

const express = require('express');
const { upflyUpload } = require('upfly');

const app = express();

app.post('/upload',
  upflyUpload({
    fields: {
      images: {
        format: 'webp',
        quality: 80
      }
    }
  }),
  (req, res) => {
    res.json({
      success: true,
      files: req.files.images
    });
  }
);

app.listen(3000);

curl -X POST -F "[email protected]" http://localhost:3000/upload

Your image is automatically converted to WebP at 80% quality. Typical savings: 30–70% file size reduction.

Cloud Upload

Upload directly to Cloudinary with automatic image optimization:

app.post('/upload',
  upflyUpload({
    fields: {
      avatar: {
        cloudStorage: true,
        cloudProvider: 'cloudinary',
        cloudConfig: {
          cloud_name: process.env.CLOUDINARY_CLOUD_NAME,
          api_key: process.env.CLOUDINARY_API_KEY,
          api_secret: process.env.CLOUDINARY_API_SECRET,
          folder: 'avatars'
        },
        format: 'webp',
        quality: 85
      }
    },
    limit: 5 * 1024 * 1024,
    safeFile: true
  }),
  (req, res) => {
    res.json({ url: req.files.avatar[0].cloudUrl });
  }
);

Install the Cloudinary SDK:

npm install cloudinary

AWS S3 and Google Cloud Storage are also supported. Each field can use a different provider. See the Cloud Setup Guide for full S3 and GCS configuration.

| Provider | Required Package | |---|---| | Cloudinary | npm install cloudinary | | AWS S3 | npm install @aws-sdk/client-s3 @aws-sdk/lib-storage | | Google Cloud Storage | npm install @google-cloud/storage |

Multi-Field Forms

Different processing rules for each field in a single upload:

app.post('/post',
  upflyUpload({
    fields: {
      // Thumbnail: aggressive compression, kept in memory
      thumbnail: {
        format: 'webp',
        quality: 60,
        output: 'memory'
      },

      // Main image: high quality, uploaded to S3
      image: {
        cloudStorage: true,
        cloudProvider: 's3',
        cloudConfig: {
          region: process.env.AWS_REGION,
          bucket: process.env.AWS_BUCKET,
          accessKeyId: process.env.AWS_ACCESS_KEY_ID,
          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
        format: 'webp',
        quality: 85
      },

      // Attachments: no conversion, saved to disk
      attachments: {
        output: 'disk',
        keepOriginal: true
      }
    },
    outputDir: './uploads',
    safeFile: true
  }),
  (req, res) => {
    res.json({
      thumbnail: req.files.thumbnail[0].buffer.toString('base64'),
      imageUrl: req.files.image[0].cloudUrl,
      attachments: req.files.attachments.map(f => f.path)
    });
  }
);

Disk Storage

Save files directly to disk without conversion — useful for documents, PDFs, and other non-image uploads:

app.post('/documents',
  upflyUpload({
    fields: {
      files: {
        output: 'disk',
        keepOriginal: true
      }
    },
    outputDir: './user-documents',
    limit: 50 * 1024 * 1024  // 50 MB
  }),
  (req, res) => {
    const files = req.files.files.map(f => ({
      path: f.path,
      name: f.originalname,
      size: f.size
    }));
    res.json({ files });
  }
);

Conversion-Only Mode

Already using Multer? Use upflyConvert to add image conversion to your existing setup without changing your upload logic:

const multer = require('multer');
const { upflyConvert } = require('upfly');

const upload = multer({ storage: multer.memoryStorage() });

app.post('/upload',
  upload.fields([{ name: 'avatar', maxCount: 1 }]),
  upflyConvert({
    fields: {
      avatar: { format: 'webp', quality: 80, output: 'disk' }
    },
    outputDir: './uploads'
  }),
  (req, res) => {
    res.json({ file: req.files.avatar[0] });
  }
);

upflyConvert requires files to be in memory (Multer's memoryStorage). It supports memory and disk output, but not cloud storage.

API Reference

upflyUpload(options)

Complete upload + conversion middleware. Handles file interception, processing, and storage.

Options:

| Property | Type | Default | Description | |---|---|---|---| | fields | Record<string, FieldConfig> | — | Field configurations keyed by HTML form field name. Required. | | outputDir | string | './uploads' | Directory for disk output. Field-level outputDir takes precedence. | | limit | number | 10485760 | Maximum file size in bytes (default: 10 MB). | | safeFile | boolean | false | Enable automatic backup stream for zero data loss. |

upflyConvert(options)

Conversion-only middleware for files already in req.file / req.files. Must follow a Multer middleware that uses memoryStorage().

Options:

| Property | Type | Default | Description | |---|---|---|---| | fields | Record<string, FieldConfig> | — | Field configurations. Required. | | outputDir | string | './uploads' | Directory for disk output. | | safeFile | boolean | false | Enable backup fallback. |

Field Configuration

Each field in the fields object accepts:

| Property | Type | Default | Description | |---|---|---|---| | output | 'memory' \| 'disk' | 'memory' | Where to store the processed file. | | outputDir | string | — | Field-specific output directory (overrides global). | | format | string | 'webp' | Target image format. | | quality | number | 80 | Compression quality, 1–100. | | keepOriginal | boolean | false | Skip conversion. Useful for non-image files. | | cloudStorage | boolean | false | Upload to cloud after processing. | | cloudProvider | 'cloudinary' \| 's3' \| 'gcs' | — | Cloud provider. Required when cloudStorage: true. Aliases: 'aws' for S3, 'google' for GCS. | | cloudConfig | object | — | Provider-specific configuration. Required when cloudStorage: true. See Cloud Setup Guide for per-provider fields. |

Supported image formats:

| | Formats | |---|---| | Input | JPEG, PNG, WebP, GIF, AVIF, TIFF, SVG, HEIF/HEIC | | Output | WebP, JPEG, PNG, AVIF, TIFF, GIF, HEIF |

File Result Object

After processing, each file in req.files.<fieldname>[] contains:

| Property | Type | Description | |---|---|---| | fieldname | string | Form field name | | originalname | string | Original filename | | mimetype | string | MIME type | | size | number | Final file size in bytes | | buffer | Buffer | File data (when output: 'memory') | | path | string | File path (when output: 'disk') | | cloudUrl | string | Public URL (when cloudStorage: true) | | cloudPublicId | string | Cloud provider file ID | | cloudProvider | string | Provider name | | _metadata | object | Error and fallback metadata |

Error Handling

Upfly never crashes your application. When safeFile: true is enabled, a backup copy of the original file is maintained during processing. If conversion or upload fails, the backup is used automatically.

All error information is captured in file._metadata:

const file = req.files.images[0];

if (file._metadata?.isSkipped) {
  // Total failure — file could not be processed at all
  console.error(file._metadata.errors.message);
}

if (file._metadata?.isBackupFallback) {
  // Conversion failed, but the original file was saved successfully
  console.warn('Used backup:', file._metadata.errors.conversion);
}

// file._metadata.isProcessed === true means success

The backup system uses smart thresholding: files under 7 MB are backed up in memory; larger files are streamed to a temp directory with automatic cleanup on process exit.

See the Error Handling Guide for the full _metadata structure and common error scenarios.

Cloud Config Validation

Validate your cloud credentials at application startup:

const { validateAllCloudConfigs } = require('upfly/src/cloud');

validateAllCloudConfigs(config.fields)
  .then(() => console.log('Cloud configs validated'))
  .catch(err => console.error('Cloud config error:', err));

TypeScript

Upfly ships with full type definitions. Cloud config types are conditional — set cloudProvider before cloudConfig to get provider-specific autocomplete:

import { upflyUpload, UpflyOptions, UpflyFile } from 'upfly';

const options: UpflyOptions = {
  fields: {
    avatar: {
      cloudStorage: true,
      cloudProvider: 'cloudinary',  // Set this first
      cloudConfig: {                // Now you get Cloudinary-specific types
        cloud_name: '...',
        api_key: '...',
        api_secret: '...'
      }
    }
  }
};

Upfly for VS Code

If you work with images outside of Express — converting, compressing, or uploading from your editor — check out the Upfly VS Code Extension.

• Right-click convert — select any image and convert to WebP, AVIF, PNG, or JPEG from the context menu
• Folder watchers — automatically convert images dropped into watched directories (recursive)
• Cloud upload — convert and upload to Cloudinary, S3, or GCS in one step
• Compression — reduce file size without changing format

Contributing

Contributions are welcome. Please open an issue before starting work on major changes.

1. Fork the repo
2. Create a feature branch: git checkout -b feature/your-feature
3. Commit your changes: git commit -m 'Add your feature'
4. Push: git push origin feature/your-feature
5. Open a Pull Request

License

MIT © Ramin

Website · Quick Start · Cloud Setup · Error Handling · API Reference · GitHub · Issues