@synet/fs-github
v1.0.0
Published
Github as filesystem abstraction following FileSystem pattern
Readme
@synet/fs-github
GitHub FileSystem Adapter - Transform GitHub repositories into powerful file storage systems with built-in version control, collaboration, and global CDN distribution.
Turn any GitHub repository into a fully functional filesystem with automatic versioning, instant global availability, and enterprise-grade security. Perfect for configuration management, documentation systems, and collaborative data storage.
Features
- Automatic Version Control: Every file change creates a Git commit with full history
- Global CDN Distribution: Files instantly available worldwide via GitHub's CDN
- Built-in Collaboration: Leverage GitHub's powerful collaboration tools
- Enterprise Security: GitHub's enterprise-grade authentication and authorization
- Branch-based Environments: Use Git branches for different environments
- Intelligent Caching: Built-in caching for optimal performance
- Commit Metadata: Rich commit messages with author information
- Free Hosting: Free storage for public repositories
- REST API Integration: Full GitHub API integration via Octokit
- TypeScript First: Complete type safety with comprehensive interfaces
Installation
# npm
npm install @synet/fs-github
GitHub Setup
Step 1: Create a Personal Access Token
Navigate to GitHub Settings:
- Go to GitHub.com → Profile → Settings
- Left sidebar → Developer settings
- Personal access tokens → Tokens (classic)
Generate New Token:
- Click Generate new token (classic)
- Name:
SYNET FS GitHub Package - Expiration: Choose based on your needs
Required Permissions:
repo (Full control of private repositories) ├── repo:status - Access commit status ├── repo_deployment - Access deployment status ├── public_repo - Access public repositories ├── repo:invite - Access repository invitations └── security_events - Read and write security eventsCopy Token:
- Click Generate token
- ⚠️ IMPORTANT: Copy immediately (won't be shown again!)
- Format:
ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Quick Link: Create Token
Step 2: Create Target Repository
Create a repository where files will be stored:
# Via GitHub CLI
gh repo create my-app-storage --private
# Or via GitHub web interface
# https://github.com/newStep 3: Verify Permissions
Test your setup:
import { GitHubFileSystem } from '@synet/fs-github';
const github = new GitHubFileSystem({
token: 'your_token_here',
owner: 'your-username',
repo: 'your-repo',
branch: 'main'
});
// Test connection
const info = github.getRepositoryInfo();
console.log(`Connected to: ${info.owner}/${info.repo}`);Quick Start
Basic Configuration
import { GitHubFileSystem } from '@synet/fs-github';
const githubFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'your-username',
repo: 'app-config',
branch: 'main',
authorName: 'Config Manager',
authorEmail: '[email protected]'
});
// Write configuration files
await githubFs.writeFile('/config/production.json', JSON.stringify({
database: { host: 'prod-db.com', port: 5432 },
api: { url: 'https://api.yourcompany.com' }
}));
// Read configuration
const config = JSON.parse(await githubFs.readFile('/config/production.json'));
// List files
const files = await githubFs.readDir('/config');
console.log('Config files:', files);Environment Variables
# .env file
GITHUB_TOKEN=ghp_your_token_here
GITHUB_OWNER=your-username
GITHUB_REPO=app-storage
GITHUB_BRANCH=mainimport { GitHubFileSystem } from '@synet/fs-github';
const githubFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: process.env.GITHUB_OWNER!,
repo: process.env.GITHUB_REPO!,
branch: process.env.GITHUB_BRANCH || 'main',
authorName: 'Application',
authorEmail: '[email protected]'
});Advanced Configuration
Multi-Environment Setup
import { GitHubFileSystem } from '@synet/fs-github';
class ConfigManager {
private environments = new Map<string, GitHubFileSystem>();
constructor() {
// Production environment (main branch)
this.environments.set('production', new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'your-company',
repo: 'app-config',
branch: 'main',
authorName: 'Production Deploy',
authorEmail: '[email protected]'
}));
// Staging environment (staging branch)
this.environments.set('staging', new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'your-company',
repo: 'app-config',
branch: 'staging',
authorName: 'Staging Deploy',
authorEmail: '[email protected]'
}));
// Development environment (dev branch)
this.environments.set('development', new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'your-company',
repo: 'app-config',
branch: 'development',
authorName: 'Dev Team',
authorEmail: '[email protected]'
}));
}
getConfig(environment: string): GitHubFileSystem {
const fs = this.environments.get(environment);
if (!fs) {
throw new Error(`Unknown environment: ${environment}`);
}
return fs;
}
async deployConfig(source: string, target: string, configPath: string): Promise<void> {
const sourceFs = this.getConfig(source);
const targetFs = this.getConfig(target);
const config = await sourceFs.readFile(configPath);
await targetFs.writeFile(configPath, config);
}
}
// Usage
const configManager = new ConfigManager();
const prodConfig = configManager.getConfig('production');
await prodConfig.writeFile('/api/settings.json', JSON.stringify({
rateLimiting: { enabled: true, maxRequests: 1000 },
features: { betaFeatures: false }
}));Documentation System
import { GitHubFileSystem } from '@synet/fs-github';
class DocumentationManager {
private docsFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'your-company',
repo: 'documentation',
branch: 'main',
authorName: 'Documentation Bot',
authorEmail: '[email protected]'
});
async publishDocs(category: string, title: string, content: string): Promise<void> {
const slug = title.toLowerCase().replace(/\s+/g, '-');
const path = `/docs/${category}/${slug}.md`;
const markdown = `# ${title}
${content}
---
*Last updated: ${new Date().toISOString()}*
*Published via @synet/fs-github*
`;
await this.docsFs.writeFile(path, markdown);
}
async getDocs(category: string): Promise<string[]> {
return await this.docsFs.readDir(`/docs/${category}`);
}
async getDocContent(category: string, slug: string): Promise<string> {
return await this.docsFs.readFile(`/docs/${category}/${slug}.md`);
}
}
// Usage
const docs = new DocumentationManager();
await docs.publishDocs('api', 'Authentication Guide', `
## Overview
Our API uses OAuth 2.0 for authentication...
## Quick Start
1. Register your application
2. Get your client credentials
3. Implement the OAuth flow
`);Configuration Versioning
import { GitHubFileSystem } from '@synet/fs-github';
class VersionedConfig {
private fs: GitHubFileSystem;
constructor() {
this.fs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'your-company',
repo: 'versioned-config',
branch: 'main',
authorName: 'Config Manager',
authorEmail: '[email protected]'
});
}
async updateConfig(
path: string,
config: any,
reason: string
): Promise<void> {
const content = JSON.stringify(config, null, 2);
const commitMessage = `Update ${path}: ${reason}`;
// The GitHub filesystem automatically creates commits
await this.fs.writeFile(path, content);
console.log(`✅ Configuration updated: ${commitMessage}`);
}
async rollbackConfig(path: string, commitSha: string): Promise<void> {
// In a real implementation, you'd use GitHub API to get file content at specific commit
const history = await this.fs.getFileHistory(path);
const targetCommit = history.find(commit => commit.sha === commitSha);
if (targetCommit) {
await this.fs.writeFile(path, targetCommit.content);
console.log(`✅ Rolled back ${path} to ${commitSha}`);
}
}
async getConfigHistory(path: string): Promise<any[]> {
return await this.fs.getFileHistory(path);
}
}API Reference
Constructor Options
interface GitHubFileSystemOptions {
token: string; // GitHub personal access token
owner: string; // Repository owner (username or organization)
repo: string; // Repository name
branch?: string; // Target branch (default: 'main')
authorName?: string; // Commit author name (default: 'GitHubFileSystem')
authorEmail?: string; // Commit author email (default: '[email protected]')
autoCommit?: boolean; // Auto-commit on writes (default: true)
}File Operations
writeFile(path: string, content: string): Promise<void>- Write file and create commitreadFile(path: string): Promise<string>- Read file contentdeleteFile(path: string): Promise<void>- Delete file and create commitexists(path: string): Promise<boolean>- Check if file exists
Directory Operations
ensureDir(path: string): Promise<void>- No-op (GitHub doesn't have directories)deleteDir(path: string): Promise<void>- Not supported (throws error)readDir(path: string): Promise<string[]>- List files with path prefix
Metadata Operations
stat(path: string): Promise<FileStats>- Get file metadatagetFileHistory(path: string): Promise<CommitInfo[]>- Get file commit historygetRepositoryInfo(): RepositoryInfo- Get repository information
Cache Operations
clearCache(): void- Clear internal file cachegetCacheStats(): CacheStats- Get cache statistics
FileStats Interface
interface FileStats {
isFile(): boolean; // Always true for GitHub files
isDirectory(): boolean; // Always false for GitHub files
isSymbolicLink(): boolean; // Always false
size: number; // File size in bytes
mtime: Date; // Last modified time (commit date)
ctime: Date; // Creation time (first commit date)
atime: Date; // Access time (same as mtime)
mode: number; // File permissions (644)
}Repository Information
interface RepositoryInfo {
owner: string; // Repository owner
repo: string; // Repository name
branch: string; // Current branch
url: string; // Repository URL
}Testing
# Run all tests
npm test
# Run tests in watch mode
npm run dev:test
# Run tests with coverage
npm run coverage
# Run demo
npm run demoTest Configuration
Create test configuration:
// test-config.ts
export const testConfig = {
token: process.env.GITHUB_TEST_TOKEN!,
owner: 'test-org',
repo: 'test-repo',
branch: 'test-branch',
authorName: 'Test Bot',
authorEmail: '[email protected]'
};Integration Tests
import { GitHubFileSystem } from '@synet/fs-github';
import { testConfig } from './test-config';
describe('GitHub FileSystem Integration', () => {
let githubFs: GitHubFileSystem;
beforeEach(() => {
githubFs = new GitHubFileSystem(testConfig);
});
afterEach(async () => {
// Clean up test files
try {
const files = await githubFs.readDir('/test');
for (const file of files) {
await githubFs.deleteFile(`/test/${file}`);
}
} catch {
// Ignore cleanup errors
}
});
it('should write and read files', async () => {
const path = '/test/integration.json';
const content = JSON.stringify({ test: 'data', timestamp: Date.now() });
await githubFs.writeFile(path, content);
const retrieved = await githubFs.readFile(path);
expect(retrieved).toBe(content);
});
it('should maintain file history', async () => {
const path = '/test/versioned.txt';
await githubFs.writeFile(path, 'Version 1');
await githubFs.writeFile(path, 'Version 2');
const history = await githubFs.getFileHistory(path);
expect(history).toHaveLength(2);
expect(history[0].message).toContain('versioned.txt');
});
});Use Cases
1. Configuration Management
const configFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'company',
repo: 'app-config',
branch: 'production'
});
// Store feature flags
await configFs.writeFile('/features.json', JSON.stringify({
enableNewDashboard: true,
betaFeatures: false,
maintenanceMode: false
}));
// Store API configurations
await configFs.writeFile('/api/endpoints.json', JSON.stringify({
user: 'https://api.company.com/users',
payments: 'https://api.company.com/payments',
analytics: 'https://analytics.company.com/events'
}));2. Documentation Publishing
const docsFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'company',
repo: 'documentation',
branch: 'main'
});
// Publish API documentation
await docsFs.writeFile('/api/users.md', `
# User API
## GET /api/users
Returns a list of users...
## POST /api/users
Creates a new user...
`);
// Update changelog
const changelog = await docsFs.readFile('/CHANGELOG.md');
const updated = `## v2.1.0 - ${new Date().toISOString().split('T')[0]}
- Added user management API
- Fixed authentication bug
${changelog}`;
await docsFs.writeFile('/CHANGELOG.md', updated);3. Content Management System
class GitHubCMS {
private fs: GitHubFileSystem;
constructor() {
this.fs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'blog',
repo: 'content',
branch: 'main',
authorName: 'CMS Bot',
authorEmail: '[email protected]'
});
}
async publishPost(slug: string, frontmatter: any, content: string): Promise<void> {
const post = `---
${Object.entries(frontmatter)
.map(([key, value]) => `${key}: ${JSON.stringify(value)}`)
.join('\n')}
---
${content}`;
await this.fs.writeFile(`/posts/${slug}.md`, post);
}
async getPosts(): Promise<string[]> {
return await this.fs.readDir('/posts');
}
async getPost(slug: string): Promise<string> {
return await this.fs.readFile(`/posts/${slug}.md`);
}
}
// Usage
const cms = new GitHubCMS();
await cms.publishPost('hello-world', {
title: 'Hello World',
date: '2025-08-10',
author: 'Admin',
tags: ['announcement', 'welcome']
}, 'Welcome to our new blog platform!');4. Backup and Sync
class GitHubBackup {
private backupFs: GitHubFileSystem;
constructor() {
this.backupFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'company',
repo: 'app-backups',
branch: 'main',
authorName: 'Backup Service',
authorEmail: '[email protected]'
});
}
async backupDatabase(data: any): Promise<void> {
const timestamp = new Date().toISOString();
const filename = `/backups/db-${timestamp.split('T')[0]}.json`;
await this.backupFs.writeFile(filename, JSON.stringify({
timestamp,
data,
metadata: {
source: 'production-db',
type: 'full-backup',
size: JSON.stringify(data).length
}
}, null, 2));
}
async listBackups(): Promise<string[]> {
return await this.backupFs.readDir('/backups');
}
async restoreBackup(filename: string): Promise<any> {
const backup = await this.backupFs.readFile(`/backups/${filename}`);
return JSON.parse(backup);
}
}5. Multi-Tenant Configuration
class TenantConfigManager {
private fs: GitHubFileSystem;
constructor() {
this.fs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'saas-company',
repo: 'tenant-configs',
branch: 'main'
});
}
async setTenantConfig(tenantId: string, config: any): Promise<void> {
const path = `/tenants/${tenantId}/config.json`;
await this.fs.writeFile(path, JSON.stringify(config, null, 2));
}
async getTenantConfig(tenantId: string): Promise<any> {
const path = `/tenants/${tenantId}/config.json`;
if (await this.fs.exists(path)) {
const content = await this.fs.readFile(path);
return JSON.parse(content);
}
return this.getDefaultConfig();
}
async listTenants(): Promise<string[]> {
return await this.fs.readDir('/tenants');
}
private getDefaultConfig() {
return {
features: { advanced: false },
limits: { users: 10, storage: '1GB' },
theme: 'default'
};
}
}Performance Optimization
Caching Strategies
class OptimizedGitHubFS {
private fs: GitHubFileSystem;
private localCache = new Map<string, { content: string; timestamp: number }>();
private cacheTTL = 5 * 60 * 1000; // 5 minutes
constructor(options: GitHubFileSystemOptions) {
this.fs = new GitHubFileSystem(options);
}
async readFile(path: string): Promise<string> {
// Check local cache first
const cached = this.localCache.get(path);
const now = Date.now();
if (cached && (now - cached.timestamp) < this.cacheTTL) {
return cached.content;
}
// Fetch from GitHub
const content = await this.fs.readFile(path);
// Update cache
this.localCache.set(path, { content, timestamp: now });
return content;
}
async writeFile(path: string, content: string): Promise<void> {
await this.fs.writeFile(path, content);
// Update cache
this.localCache.set(path, { content, timestamp: Date.now() });
}
clearLocalCache(): void {
this.localCache.clear();
}
}Batch Operations
class BatchGitHubOperations {
private fs: GitHubFileSystem;
constructor(options: GitHubFileSystemOptions) {
this.fs = new GitHubFileSystem(options);
}
async batchWriteFiles(files: Array<{ path: string; content: string }>): Promise<void> {
// Process in batches to avoid rate limiting
const batchSize = 5;
const batches = [];
for (let i = 0; i < files.length; i += batchSize) {
batches.push(files.slice(i, i + batchSize));
}
for (const batch of batches) {
await Promise.all(
batch.map(file => this.fs.writeFile(file.path, file.content))
);
// Small delay between batches to respect rate limits
await new Promise(resolve => setTimeout(resolve, 1000));
}
}
async batchReadFiles(paths: string[]): Promise<Map<string, string>> {
const results = new Map<string, string>();
const readPromises = paths.map(async (path) => {
try {
const content = await this.fs.readFile(path);
results.set(path, content);
} catch (error) {
console.warn(`Failed to read ${path}:`, error);
}
});
await Promise.all(readPromises);
return results;
}
}Error Handling
Common Error Patterns
import { GitHubFileSystem } from '@synet/fs-github';
class ResilientGitHubFS {
private fs: GitHubFileSystem;
private maxRetries = 3;
private retryDelay = 1000;
constructor(options: GitHubFileSystemOptions) {
this.fs = new GitHubFileSystem(options);
}
async writeFileWithRetry(path: string, content: string): Promise<void> {
let lastError: Error;
for (let attempt = 1; attempt <= this.maxRetries; attempt++) {
try {
await this.fs.writeFile(path, content);
return; // Success
} catch (error: any) {
lastError = error;
// Don't retry authentication errors
if (error.status === 401 || error.status === 403) {
throw error;
}
// Don't retry if it's the last attempt
if (attempt === this.maxRetries) {
break;
}
// Wait before retrying
await this.delay(this.retryDelay * attempt);
}
}
throw new Error(`Failed to write file after ${this.maxRetries} attempts: ${lastError.message}`);
}
async readFileWithFallback(path: string, fallbackContent?: string): Promise<string> {
try {
return await this.fs.readFile(path);
} catch (error: any) {
if (error.status === 404 && fallbackContent !== undefined) {
return fallbackContent;
}
throw error;
}
}
private delay(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}
}
// Error handling patterns
try {
await githubFs.writeFile('/config.json', JSON.stringify(config));
} catch (error: any) {
if (error.status === 401) {
console.error('Authentication failed - check your token');
} else if (error.status === 403) {
console.error('Permission denied - check token permissions');
} else if (error.status === 404) {
console.error('Repository not found - check owner/repo');
} else if (error.status === 422) {
console.error('Validation failed - check file content');
} else {
console.error('Unexpected error:', error.message);
}
}Rate Limiting
class RateLimitedGitHubFS {
private fs: GitHubFileSystem;
private requestQueue: Array<() => Promise<any>> = [];
private processing = false;
private requestsPerMinute = 60; // GitHub rate limit
private requestInterval = 60000 / this.requestsPerMinute;
constructor(options: GitHubFileSystemOptions) {
this.fs = new GitHubFileSystem(options);
}
async queueRequest<T>(operation: () => Promise<T>): Promise<T> {
return new Promise((resolve, reject) => {
this.requestQueue.push(async () => {
try {
const result = await operation();
resolve(result);
} catch (error) {
reject(error);
}
});
this.processQueue();
});
}
private async processQueue(): Promise<void> {
if (this.processing || this.requestQueue.length === 0) {
return;
}
this.processing = true;
while (this.requestQueue.length > 0) {
const operation = this.requestQueue.shift()!;
await operation();
// Wait between requests to respect rate limits
if (this.requestQueue.length > 0) {
await this.delay(this.requestInterval);
}
}
this.processing = false;
}
private delay(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}
}Security Best Practices
Token Management
// ✅ Good: Use environment variables
const githubFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'company',
repo: 'secure-config'
});
// ❌ Avoid: Hardcoding tokens
const insecureFs = new GitHubFileSystem({
token: 'ghp_hardcoded_token_here', // Never do this!
owner: 'company',
repo: 'config'
});
// ✅ Good: Validate environment variables
function validateEnvironment(): void {
const required = ['GITHUB_TOKEN', 'GITHUB_OWNER', 'GITHUB_REPO'];
const missing = required.filter(key => !process.env[key]);
if (missing.length > 0) {
throw new Error(`Missing required environment variables: ${missing.join(', ')}`);
}
}
// ✅ Good: Use scoped tokens
// Create tokens with minimal required permissions
// Separate tokens for different environmentsAccess Control
class SecureConfigManager {
private fs: GitHubFileSystem;
private allowedPaths: string[] = ['/config/', '/settings/', '/features/'];
constructor() {
validateEnvironment();
this.fs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: process.env.GITHUB_OWNER!,
repo: process.env.GITHUB_REPO!
});
}
async writeFile(path: string, content: string): Promise<void> {
this.validatePath(path);
this.validateContent(content);
await this.fs.writeFile(path, content);
}
async readFile(path: string): Promise<string> {
this.validatePath(path);
return await this.fs.readFile(path);
}
private validatePath(path: string): void {
const isAllowed = this.allowedPaths.some(allowed => path.startsWith(allowed));
if (!isAllowed) {
throw new Error(`Access denied: Path '${path}' is not allowed`);
}
}
private validateContent(content: string): void {
try {
JSON.parse(content);
} catch {
throw new Error('Invalid JSON content');
}
if (content.includes('password') || content.includes('secret')) {
throw new Error('Sensitive data detected in content');
}
}
}Development
Building
npm run buildLinting
npm run lint
npm run lint:fixFormatting
npm run formatDemo
npm run demoTroubleshooting
Common Issues
| Error | Cause | Solution |
|-------|-------|----------|
| 401 Unauthorized | Invalid token | Check token validity and regenerate if needed |
| 403 Forbidden | Insufficient permissions | Ensure token has repo scope |
| 404 Not Found | Repository doesn't exist | Create repository or check owner/repo names |
| 422 Unprocessable Entity | Invalid file content | Check file content and encoding |
| 409 Conflict | Concurrent modifications | Implement retry logic with exponential backoff |
Debug Mode
// Enable debug logging
process.env.DEBUG = 'github-fs:*';
const githubFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: 'test-owner',
repo: 'test-repo'
});
// Operations will now log detailed information
await githubFs.writeFile('/debug.txt', 'test content');Health Check
async function healthCheck(): Promise<boolean> {
try {
const githubFs = new GitHubFileSystem({
token: process.env.GITHUB_TOKEN!,
owner: process.env.GITHUB_OWNER!,
repo: process.env.GITHUB_REPO!
});
// Test basic operations
const testPath = '/.health-check';
const testContent = JSON.stringify({ timestamp: Date.now() });
await githubFs.writeFile(testPath, testContent);
const retrieved = await githubFs.readFile(testPath);
await githubFs.deleteFile(testPath);
return retrieved === testContent;
} catch (error) {
console.error('Health check failed:', error);
return false;
}
}License
MIT License - see LICENSE file for details.
Contributing
Contributions are welcome! Please read our Contributing Guide for details on our code of conduct and the process for submitting pull requests.
Development Setup
git clone https://github.com/synthetism/fs-github.git
cd fs-github
npm install
# Set up test environment
cp .env.example .env
# Edit .env with your test GitHub token and repository
npm testRelated Packages
- @synet/fs - Core filesystem abstraction and Unit Architecture
- @synet/fs-azure - Azure Blob Storage adapter
- @synet/fs-gcs - Google Cloud Storage adapter
- @synet/fs-s3 - AWS S3 storage adapter
- @synet/fs-linode - Linode Object Storage adapter
- @synet/fs-memory - In-memory storage adapter
Built with ❤️ by the Synet Team
