Project Background

This project can be using in an existing app that has an Express backend file server or it can act as a standalone file server. It's designed for use with standard ISoft-style-application MYSQL schema(SELECTs from file and filechunk tables).

Installing

npm i @isoftdata/file-service

Getting Started

This project is designed with two use cases in mind:

1. For use in a project with an existing web server and database connection/pool
2. As a standalone web server

For Use in a Project with an Existing Web Server

API

@isoftdata/file-service exports an object with a registerFileServiceRoute function on it. That function takes the an object with the following shape:

| Name | Type | Description | Default Value | | ----------- | ---------- | ------------------------------------------------------------------------------------------------------------ | ------------------------ | | webServer | Object | Must be an express server instance | N/A - Prop is Required | | connection | Object | Must be one of mysql's Pool, Connection, or PoolConnection classes | N/A - Prop is Required | | routePrefix | String | The HTTP route from which the image(s) will be served | /* |

connection Tip: if you give a Pool class for the connection, a single connection will be pulled from the pool for each individual request and released at the end of that request. The connection won't be released if you pass a PoolConnection or Connection(of course).

routePrefix Tip: if you want to serve images from the root of the web server, give an empty string for routePrefix. Note that /* will not match a root-level URL, so the empty string is the way to do this.

⚠️ routePrefix must not contain **. It is interpolated into the express route pattern, and /** compiles to a regex that backtracks catastrophically — a real download URL never finishes matching, so the request hangs forever holding a pooled connection until the pool is exhausted. /** was the previous default and is now rejected at startup with an error rather than allowed to hang. Use /*, which matches the same URLs.

Example usage:

import express from 'express'
import fileUpload from 'express-fileupload'
import mysql from 'mysql'
import { registerFileServiceRoute } from '@isoftdata/file-service'

const expressServer = express()
const port = process.env.PORT ?? 80

const pool = mysql.createPool({
	host: 'example.org',
	user: 'bob',
	password: 'secret',
	database: 'my_db',
})

// This will enable file upload support for the express server.
expressServer.use(fileUpload())

// This adds the `/file` and `/upload` routes to the express server.
registerFileServiceRoute({ webServer: expressServer, connection: pool })

// Start the server
expressServer.listen({ port }, () => console.log(`🚀 Server ready on port ${port}!`))

As a Standalone Web Server

1. Clone this repository
2. Run npm i
3. Create a .env file in the root of the project that looks like this, but with valid values filled out:

MYSQL_HOST=
MYSQL_PORT=      # optional, defaults to 3306
MYSQL_USER=
MYSQL_PASSWORD=
MYSQL_DATABASE=
PORT=

MYSQL_PORT may be left blank to use the driver default of 3306. If you do set it, it must be a whole number from 1 to 65535 — anything else fails at startup rather than silently falling back and timing out against the wrong port.

MYSQL_PASSWORD can also be supplied as a Docker secret at /run/secrets/mysql_password, which takes effect when the environment variable is unset.

4. Run npm run start

The web server will accept requests to any path so long as the path ends in <md5-checksum-of-the-file><file-id>/<file-name>.

In other words, foo/bar/baz/fd90a7512022dd825fcd41982ec466931956/mySweetPicture.jpg is just as valid and the same as images/fd90a7512022dd825fcd41982ec466931956/mySweetPicture.jpg.

Request Parameters

When making a request for an image, a few GET parameters are available(all optional):

1. width
2. height
3. background (defaults to 255)
4. fit

These parameters, if valid, are passed into the Sharp image processor. For details on these parameters, check out the Sharp resize docs.

Using the Client

The FileServiceClient class is exported by this module and can be used to upload images. Example usage:

import { FileServiceClient } from '@isoftdata/file-service'
import fs from 'fs'

const client = new FileServiceClient({
	maxImageHeight: 1024, // optional
	maxImageWidth: 1024, // optional
	url: 'http://localhost:4000', // wherever the service is
})

client.addFile({
	name: 'myFile.jpg', // optional
	stream: fs.createReadStream('./myFile.jpg'),
})

client.addFile({
	// Safe to omit `name` for an fs.ReadStream: the filename is taken from the stream's path,
	// so this is stored as 'myFile.jpg' with a Content-Type of image/jpeg.
	stream: fs.createReadStream('./myFile.jpg'),
})

client.addFile({
	// But any OTHER Readable has no filename to recover, so omitting `name` here means the
	// file is stored as 'blob' with no usable type — see the warning below.
	stream: anyReadableStream,
})

// This will upload all the files that have been added above. It will return a Promise<UploadResult[]>
const resultsPromise = client.upload()

⚠️ Pass name for anything that isn't an fs.ReadStream. The part's Content-Type is derived from the filename, and the server decides whether a file is an image from that type. With no name and no recoverable path, a file is stored as blob with application/octet-stream, and then:

• maxImageHeight / maxImageWidth are ignored, so the image is stored full-size
• ?width= / ?height= are ignored on download
• no Content-Type header is sent when the file is served

None of this reports an error — the upload still returns success: true.

What about authentication?

The file service doesn't have any concept or opinions about authentication requirements baked in. If you want to ensure that your files can't be accessed by someone that could guess the MD5 hash + fileid combo, we recommend you write an authentication checking middleware function that you register before the file service. Something like this:

import express from 'express'
import cookieParser from 'cookie-parser'
const app = express()
expressServer.use(cookieParser())

expressServer.use('/files', async (req, res, next) => {
	const authenticated = await isAuthenticated(req.cookies.token)

	if (!authenticated) {
		next(new Error('Not Authenticated'))
	} else {
		next()
	}
})

where the isAuthenticated function is from your app and knows how to check if a session is valid/authenticated.

What schema is required?

Some common ISoft schema is expected:

Tables:

• file
• filechunk

Functions:

• f_get_attachment_data

Running tests

npm test                  # unit tests — no database or running service needed
npm run test:integration  # live tests against a running service (see below)

Tests are split by directory: test/unit/ runs on npm test, test/integration/ only on npm run test:integration. Both scripts glob their directory, so a new test/unit/whatever.test.ts is picked up with no change to package.json.

Both are compiled first — a pretest step builds tsconfig.test.json to dist-test/ and the runner executes the emitted .js, so tests run against the same output consumers get. That build is separate from npm run build on purpose: compiling test/ needs a rootDir covering it, which would move published output from dist/entry/ to dist/src/entry/ and break consumers. dist-test/ is gitignored and safe to delete.

Integration tests

The integration suite needs a real service and database. Docker can provide both:

npm run test:docker       # throwaway MySQL + the service from your working tree
npm run test:docker:full  # throwaway MySQL + the service built from the Dockerfile

Either command brings the stack up, runs the suite, and tears it down again — including when the tests fail, so nothing is left holding port 4000 or 3307. Use the first while developing, since it runs the code in your working tree; the second matches how this actually deploys and is the better pre-merge check.

To iterate without rebuilding each time:

npm run docker:db:up      # start just the database (host port 3307)
npm run start:test-db     # build and run the service against it, in the foreground
npm run test:integration  # in another terminal, as often as you like
npm run docker:db:down    # stop the database and delete its data

The throwaway database is created from test/docker/schema.sql, which is the minimum this service needs — the file and filechunk tables plus f_get_attachment_data — not the full ISoft schema. It has no volume, so docker:db:down discards everything.

To point the suite at a service you are already running, set FILE_SERVICE_URL.

The suite skips rather than fails when nothing is listening, so a green npm run test:integration does not by itself mean those tests ran. Check the output for skipping live upload tests before treating it as coverage — npm run test:docker avoids the trap by guaranteeing something is there.

Breaking changes

Version 3

The only breaking change from version 2 to 3 was the requirement of Node 16

Version 4

Removed opinion of the File service that only images can be uploaded using the imported File Service functions. *Chunker chunks files properly

Version 5

Updated from CJS to ESM

Version 6

Supports MySQL and MySQL2 through the abstraction layer of the @isoftdata/utility-db pool/connection wrappers.

Supports/requires/uses Node 20 in Docker builds and therefore requires Ubuntu 20 or higher on the host OS.

Version 7

Drops support for Node versions < 20.6.0 and drops dotenv dependency in favor of native process.loadEnvFile and the --env-file-if-exists CLI flag.

The default routePrefix changed from /** to /*, and a prefix containing ** is now rejected outright. The old default compiled to a catastrophically backtracking regex, so every download request hung forever while holding a pooled connection — see the routePrefix warning above.

If you were passing the old default explicitly, the service now throws at startup instead of hanging. Change that value to /*, which matches the same URLs.

FileServiceClient no longer relies on form-data, so a part's filename and Content-Type come from name — or, for an fs.ReadStream, from the stream's path. Uploading any other Readable without a name now stores the file as blob with no usable type, which silently disables image resizing; see the client warning above.