github-filesystem

A TypeScript library that allows you to interact with GitHub repositories like a file system. Supports both instant mode (direct writes to GitHub) and commit mode (stage changes in Upstash KV before committing).

Features

• 🚀 Instant Mode: Write directly to GitHub on each operation
• 📦 Commit Mode: Stage multiple changes and commit them together
• 💾 Resume Work: Resume interrupted work sessions from Upstash KV
• 🔄 Full File System API: writeFile, readFile, readdir, mkdir, deleteFile
• 📁 Node.js fs Compatible: Drop-in replacement for fs.promises API
• 🔁 Callback Support: Works with both promises and callbacks
• ⏰ Auto-expiry: Work sessions automatically expire after 30 days

Installation

npm install github-filesystem

Prerequisites

You'll need:

1. A GitHub personal access token with repo permissions
2. An Upstash Redis database (for commit mode)

Important: Credentials are read from environment variables:

• GITHUB_TOKEN - GitHub personal access token
• UPSTASH_REDIS_REST_URL - Upstash Redis REST URL
• UPSTASH_REDIS_REST_TOKEN - Upstash Redis REST token

Usage

Basic Setup

import { GitHubFS } from "github-filesystem";

// Credentials are automatically read from environment variables:
// - GITHUB_TOKEN
// - UPSTASH_REDIS_REST_URL
// - UPSTASH_REDIS_REST_TOKEN

const fs = new GitHubFS({
  repo: "owner/repo-name",
  namespace: "my-project", // Used to organize work sessions
  branch: "main", // Optional, defaults to "main"
});

Instant Mode

In instant mode, every operation immediately writes to GitHub:

// Write a file (creates a commit)
await fs.writeFile("docs/README.md", "# Hello World");

// Read a file
const content = await fs.readFile("docs/README.md");
console.log(content.toString());

// List directory contents
const entries = await fs.readdir("docs");
entries.forEach(entry => {
  console.log(`${entry.name} (${entry.type})`);
});

// Create a directory
await fs.mkdir("src/components", { recursive: true });

// Delete a file (creates a commit)
await fs.deleteFile("old-file.txt");

Commit Mode

In commit mode, changes are staged in Upstash KV and committed together:

// Start a work session
const sessionId = await fs.startWork();
console.log(`Started work session: ${sessionId}`);

// Make multiple changes (all staged in KV)
await fs.writeFile("file1.txt", "Content 1");
await fs.writeFile("file2.txt", "Content 2");
await fs.mkdir("new-folder", { recursive: true });
await fs.deleteFile("old-file.txt");

// Read operations check KV first, then GitHub
const content = await fs.readFile("file1.txt");

// Commit all changes as a single commit
await fs.commitWork("Add multiple files and reorganize structure");

Resume Work

Resume an interrupted work session:

// Try to resume the last work session for this namespace
const sessionId = await fs.resumeWork();

if (sessionId) {
  console.log(`Resumed work session: ${sessionId}`);
  
  // Continue making changes
  await fs.writeFile("another-file.txt", "More content");
  
  // Commit when done
  await fs.commitWork("Complete the interrupted work");
} else {
  console.log("No previous work session found");
}

Cancel Work

Cancel a work session without committing:

await fs.startWork();

// Make some changes...
await fs.writeFile("temp.txt", "Temporary content");

// Decide not to commit
await fs.cancelWork(); // All staged changes are discarded

Node.js fs Compatible API

GitHubFS implements a Node.js fs-compatible API, making it a drop-in replacement for most file system operations (except for startWork/commitWork/cancelWork which are GitHubFS-specific).

fs.promises API (Recommended)

All methods return Promises and work exactly like Node.js fs.promises:

import { GitHubFS } from "github-filesystem";

const fs = new GitHubFS({ repo: "owner/repo" });

// Read/write files
await fs.readFile("file.txt");
await fs.writeFile("file.txt", "content");
await fs.appendFile("file.txt", "more content");

// File operations
await fs.copyFile("src.txt", "dest.txt");
await fs.rename("old.txt", "new.txt");
await fs.unlink("file.txt");

// Directory operations
await fs.readdir("path/to/dir");
await fs.mkdir("new-dir", { recursive: true });
await fs.rmdir("dir", { recursive: true });
await fs.rm("file-or-dir", { recursive: true, force: true });

// File info
const stats = await fs.stat("file.txt");
console.log(stats.isFile(), stats.isDirectory(), stats.size);

await fs.access("file.txt"); // Check if file exists
const exists = await fs.exists("file.txt"); // Returns boolean

Callback-based API

For compatibility with older code, callback-based methods are also available:

// Read file with callback
fs.readFileCallback("file.txt", (err, data) => {
  if (err) throw err;
  console.log(data.toString());
});

// Write file with callback
fs.writeFileCallback("file.txt", "content", (err) => {
  if (err) throw err;
  console.log("File written!");
});

// Read directory with callback
fs.readdirCallback("path", (err, files) => {
  if (err) throw err;
  console.log(files);
});

// Stat with callback
fs.statCallback("file.txt", (err, stats) => {
  if (err) throw err;
  console.log(stats.isFile());
});

// Other callback methods:
// mkdirCallback, unlinkCallback, rmdirCallback, existsCallback

Synchronous API (Not Supported)

Synchronous operations are not supported for remote filesystems and will throw an error:

try {
  fs.readFileSync("file.txt"); // Throws error
} catch (error) {
  console.error("Synchronous operations are not supported");
}

Using as a Drop-in Replacement

You can use GitHubFS as a drop-in replacement for fs.promises:

// Before (using Node.js fs)
import { promises as fs } from "fs";

// After (using GitHubFS)
import { GitHubFS } from "github-filesystem";
const fs = new GitHubFS({ repo: "owner/repo" });

// All your existing code works the same!
const content = await fs.readFile("file.txt");
await fs.writeFile("output.txt", content);
const files = await fs.readdir(".");

Note: The only difference is you need to use startWork()/commitWork() if you want to batch changes, otherwise each operation creates an immediate commit.

API Reference

Constructor

new GitHubFS(config: GitHubFSConfig)

Config Options:

• repo: GitHub repository in format "owner/repo"
• namespace: Namespace for organizing work sessions
• branch: Git branch to work with (default: "main")

Environment Variables (Required):

• GITHUB_TOKEN: GitHub personal access token
• UPSTASH_REDIS_REST_URL: Upstash Redis REST URL
• UPSTASH_REDIS_REST_TOKEN: Upstash Redis REST token

Mode Management

startWork(): Promise<string>

Start a work session (commit mode). Returns the session ID.

resumeWork(): Promise<string | null>

Resume the last work session for this namespace. Returns the session ID if found, null otherwise.

commitWork(message: string): Promise<void>

Commit all staged changes to GitHub with the given commit message.

cancelWork(): Promise<void>

Cancel the current work session without committing changes.

getMode(): FSMode

Get the current mode ("instant" or "commit").

getCurrentSessionId(): string | null

Get the current work session ID (if in commit mode).

File Operations (fs.promises compatible)

readFile(path: string, options?: ReadFileOptions): Promise<Buffer>

Read a file. In commit mode, checks KV first, then falls back to GitHub.

writeFile(path: string, content: Buffer | string, options?: WriteFileOptions): Promise<void>

Write a file. In instant mode, creates a commit immediately. In commit mode, stages the change in KV.

appendFile(path: string, data: string | Buffer, options?: WriteFileOptions): Promise<void>

Append to a file. Creates the file if it doesn't exist.

copyFile(src: string, dest: string): Promise<void>

Copy a file from source to destination.

rename(oldPath: string, newPath: string): Promise<void>

Rename or move a file.

unlink(path: string): Promise<void>

Delete a file. Alias for deleteFile().

deleteFile(path: string): Promise<void>

Delete a file. In instant mode, creates a commit immediately. In commit mode, stages the deletion.

readdir(path: string, options?: ReaddirOptions): Promise<DirEntry[] | string[]>

List directory contents. Returns an array of entries with name, type, path, and optional sha/size.

Options:

• withFileTypes: Return Dirent objects instead of strings (default: false)

mkdir(path: string, options?: MkdirOptions): Promise<void>

Create a directory. Since GitHub doesn't support empty directories, this creates a .gitkeep file.

Options:

• recursive: Create parent directories if they don't exist (default: false)

rmdir(path: string, options?: { recursive?: boolean }): Promise<void>

Remove a directory. Alias for rm().

rm(path: string, options?: RmOptions): Promise<void>

Remove a file or directory.

Options:

• recursive: Remove directories and their contents recursively (default: false)
• force: Ignore errors if file doesn't exist (default: false)

stat(path: string): Promise<Stats>

Get file or directory statistics. Returns a Stats object with isFile(), isDirectory(), size, etc.

lstat(path: string): Promise<Stats>

Get file or directory statistics (alias for stat(), no symlink support).

access(path: string, mode?: number): Promise<void>

Check if a file exists and is accessible. Throws if not accessible.

exists(path: string): Promise<boolean>

Check if a file or directory exists. Returns true or false.

Callback-based Methods

All promise-based methods have callback equivalents:

• readFileCallback(path, [options], callback)
• writeFileCallback(path, data, [options], callback)
• readdirCallback(path, [options], callback)
• statCallback(path, callback)
• mkdirCallback(path, [options], callback)
• unlinkCallback(path, callback)
• rmdirCallback(path, [options], callback)
• existsCallback(path, callback)

Synchronous Methods (Not Supported)

These methods throw an error:

• readFileSync(), writeFileSync(), readdirSync(), statSync(), mkdirSync(), unlinkSync(), rmdirSync(), existsSync()

How It Works

Instant Mode

• Each operation directly interacts with the GitHub API
• Every write/delete creates a new commit
• Simple but creates many commits

Commit Mode

• Changes are stored in Upstash KV with a 30-day expiry
• KV keys are prefixed with {namespace}/{sessionId}/{filepath}
• Session metadata tracks modified and deleted files
• commitWork() creates a single commit with all changes
• Read operations check KV first (for staged changes), then GitHub

Resume Functionality

• Session metadata is stored at {namespace}/session
• resumeWork() retrieves the last session for the namespace
• Allows continuing work after interruptions or across different environments

Examples

Example 1: Batch Updates

const fs = new GitHubFS({ /* config */ });

await fs.startWork();

// Update multiple documentation files
const files = ["README.md", "CONTRIBUTING.md", "LICENSE"];
for (const file of files) {
  const content = await fs.readFile(file);
  const updated = content.toString().replace("2023", "2024");
  await fs.writeFile(file, updated);
}

await fs.commitWork("Update copyright year to 2024");

Example 2: Safe Experimentation

const fs = new GitHubFS({ /* config */ });

await fs.startWork();

try {
  // Try some changes
  await fs.writeFile("config.json", JSON.stringify(newConfig));
  await fs.deleteFile("old-config.json");
  
  // Test if everything works...
  const valid = await validateConfig();
  
  if (valid) {
    await fs.commitWork("Update configuration");
  } else {
    await fs.cancelWork(); // Rollback
  }
} catch (error) {
  await fs.cancelWork(); // Rollback on error
  throw error;
}

Example 3: Long-Running Tasks

const fs = new GitHubFS({ /* config */ });

// Start or resume work
let sessionId = await fs.resumeWork();
if (!sessionId) {
  sessionId = await fs.startWork();
}

// Process files incrementally
for (const file of largeFileList) {
  await processAndWrite(fs, file);
  
  // Can stop and resume later...
}

await fs.commitWork("Process all files");

Environment Variables

Credentials are automatically loaded from environment variables. Set them in your .env file or environment:

# .env file
GITHUB_TOKEN=ghp_your_github_token
UPSTASH_REDIS_REST_URL=https://your-redis.upstash.io
UPSTASH_REDIS_REST_TOKEN=your_upstash_token

Then use the class:

const fs = new GitHubFS({
  repo: process.env.GITHUB_REPO || "owner/repo",
  namespace: process.env.NAMESPACE || "default",
});

Error Handling

try {
  await fs.readFile("non-existent.txt");
} catch (error) {
  console.error("File not found:", error);
}

try {
  await fs.startWork();
  await fs.startWork(); // Error: already in commit mode
} catch (error) {
  console.error("Cannot start work:", error);
}

Limitations

• GitHub API rate limits apply (5000 requests/hour for authenticated requests)
• Maximum file size: 100 MB (GitHub limit)
• Upstash KV expiry: 30 days maximum
• Empty directories are not supported (GitHub limitation) - use .gitkeep files

License

MIT

Contributing

Contributions are welcome! Please open an issue or submit a pull request.