@synet/fs-gcs

Google Cloud Storage adapter for @synet/fs

Store files in Google Cloud Storage with the familiar filesystem interface. Perfect for GCP-native applications that need scalable cloud storage with automatic authentication and intelligent caching.

Installation

npm install @synet/fs-gcs @synet/fs

Quick Start

import { AsyncFileSystem } from '@synet/fs';
import { GCSFileSystem } from '@synet/fs-gcs';

// Create GCS adapter
const gcsAdapter = new GCSFileSystem({
  projectId: 'my-gcp-project',
  bucketName: 'my-storage-bucket',
  keyFilename: './path/to/service-account.json' // Optional
});

// Use with AsyncFileSystem
const fs = AsyncFileSystem.create({ adapter: gcsAdapter });

// Standard filesystem operations
await fs.writeFile('config.json', JSON.stringify(config));
const data = await fs.readFile('config.json');
const files = await fs.readDir('uploads/');
const exists = await fs.exists('document.pdf');

Configuration

interface GCSFileSystemOptions {
  projectId: string;           // GCP project ID
  bucketName: string;          // GCS bucket name
  keyFilename?: string;        // Path to service account key file
  credentials?: object;        // Service account credentials object
  prefix?: string;            // Optional: namespace all files
  location?: string;          // Bucket location (default: 'US')
  storageClass?: string;      // Storage class (STANDARD, NEARLINE, etc.)
}

Usage Examples

Basic Operations

import { AsyncFileSystem } from '@synet/fs';
import { GCSFileSystem } from '@synet/fs-gcs';

const gcsAdapter = new GCSFileSystem({
  projectId: 'my-gcp-project',
  bucketName: 'documents',
  keyFilename: './service-account-key.json'
});

const fs = AsyncFileSystem.create({ adapter: gcsAdapter });

// File operations
await fs.writeFile('reports/monthly.pdf', pdfData);
const report = await fs.readFile('reports/monthly.pdf');

// Directory operations
await fs.ensureDir('uploads');
const uploadedFiles = await fs.readDir('uploads');

// Check existence
if (await fs.exists('config/app.json')) {
  const config = JSON.parse(await fs.readFile('config/app.json'));
}

With Prefix (Namespacing)

const gcsAdapter = new GCSFileSystem({
  projectId: 'my-gcp-project',
  bucketName: 'shared-bucket',
  keyFilename: './service-account-key.json',
  prefix: 'myapp/' // All files prefixed with 'myapp/'
});

const fs = AsyncFileSystem.create({ adapter: gcsAdapter });

// Files stored as: myapp/config.json, myapp/data/users.json
await fs.writeFile('config.json', configData);
await fs.writeFile('data/users.json', userData);

Using Credentials Object

const gcsAdapter = new GCSFileSystem({
  projectId: 'my-gcp-project',
  bucketName: 'mycontainer',
  credentials: {
    type: 'service_account',
    project_id: 'my-gcp-project',
    private_key_id: 'key-id',
    private_key: process.env.GCP_PRIVATE_KEY,
    client_email: '[email protected]',
    client_id: 'client-id',
    auth_uri: 'https://accounts.google.com/o/oauth2/auth',
    token_uri: 'https://oauth2.googleapis.com/token'
  }
});

Application Default Credentials

// Uses Application Default Credentials (ADC)
const gcsAdapter = new GCSFileSystem({
  projectId: 'my-gcp-project',
  bucketName: 'mycontainer'
  // No keyFilename or credentials - uses ADC
});

Custom Storage Class

const gcsAdapter = new GCSFileSystem({
  projectId: 'my-gcp-project',
  bucketName: 'archive-data',
  storageClass: 'COLDLINE', // For archival data
  location: 'europe-west1'
});

Multi-Layer Composition

Combine with other filesystem layers for advanced functionality:

import { AsyncFileSystem } from '@synet/fs';
import { GCSFileSystem } from '@synet/fs-gcs';
import { 
  AsyncObservableFileSystem,
  AsyncCachedFileSystem,
  AsyncWithIdFileSystem 
} from '@synet/fs';

const gcsAdapter = new GCSFileSystem(gcsConfig);

const fs = AsyncFileSystem.create({
  adapter: new AsyncObservableFileSystem(    // Event monitoring
    new AsyncCachedFileSystem(               // Intelligent caching
      new AsyncWithIdFileSystem(gcsAdapter)   // Deterministic IDs
    )
  )
});

// Now you have: GCS storage + caching + observability + ID mapping

Environment Variables

# Option 1: Service account key file
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
GCP_PROJECT_ID=my-gcp-project

# Option 2: Service account key as JSON string
GCP_SERVICE_ACCOUNT_KEY='{"type":"service_account","project_id":"..."}'
GCP_PROJECT_ID=my-gcp-project

# Option 3: Individual credentials
GCP_PROJECT_ID=my-gcp-project
GCP_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
[email protected]

Authentication Methods

1. Service Account Key File (Recommended for Development)

const gcsAdapter = new GCSFileSystem({
  projectId: process.env.GCP_PROJECT_ID,
  bucketName: 'mybucket',
  keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS
});

2. Application Default Credentials (Recommended for Production)

// Works on GCE, GKE, Cloud Run, Cloud Functions
const gcsAdapter = new GCSFileSystem({
  projectId: process.env.GCP_PROJECT_ID,
  bucketName: 'mybucket'
  // ADC automatically detected
});

3. Explicit Credentials Object

const gcsAdapter = new GCSFileSystem({
  projectId: process.env.GCP_PROJECT_ID,
  bucketName: 'mybucket',
  credentials: JSON.parse(process.env.GCP_SERVICE_ACCOUNT_KEY)
});

Testing

import { AsyncFileSystem } from '@synet/fs';
import { GCSFileSystem } from '@synet/fs-gcs';
import { MemFileSystem } from '@synet/fs-memory';

// Production
const gcsFs = AsyncFileSystem.create({ 
  adapter: new GCSFileSystem(gcsConfig) 
});

// Testing - same interface, different storage
const testFs = AsyncFileSystem.create({ 
  adapter: new MemFileSystem() 
});

// Your service works with both
class DocumentService {
  constructor(private fs: AsyncFileSystem) {}
  
  async saveDocument(id: string, content: string) {
    await this.fs.writeFile(`documents/${id}.txt`, content);
  }
}

Features

• Google Cloud Storage Integration: Direct integration with GCS APIs
• Automatic Authentication: Support for ADC, service accounts, and key files
• Bucket Management: Automatic bucket creation and configuration
• Storage Classes: Support for all GCS storage classes (Standard, Nearline, Coldline, Archive)
• Global CDN: Leverage Google's global edge network
• IAM Integration: Full Google Cloud IAM support
• Lifecycle Management: Automatic object lifecycle policies
• Versioning Support: Object versioning and retention policies
• Prefix Support: Namespace files within buckets
• Error Handling: Comprehensive GCS-specific error handling
• TypeScript: Full TypeScript support with proper types

Error Handling

try {
  await fs.readFile('nonexistent.txt');
} catch (error) {
  if (error.code === 404) {
    console.log('File does not exist in Google Cloud Storage');
  } else if (error.code === 403) {
    console.log('Permission denied - check IAM permissions');
  }
}

IAM Permissions

Your service account needs these minimal permissions:

{
  "bindings": [
    {
      "role": "roles/storage.objectAdmin",
      "members": [
        "serviceAccount:[email protected]"
      ]
    }
  ]
}

Performance Tips

• Use ADC in production for better security and automatic credential rotation
• Enable bucket versioning for critical data
• Use appropriate storage classes based on access patterns
• Implement caching layer for frequently accessed files
• Use prefix organization for better performance with large datasets

License

MIT

Part of the @synet/fs ecosystem.