http-ifs

What is HTTP-IFS?

HTTP-IFS (HTTP In-Memory Filesystem) is a lightweight library that lets you read files directly from HTTP/HTTPS URLs without downloading them to disk. It's like having a virtual filesystem that streams data from the web directly into your memory.

Unlike traditional download methods, HTTP-IFS uses HTTP Range Requests to fetch only the parts you need, caches data intelligently in memory, and supports streaming to other commands.

Why HTTP-IFS?

• Zero Disk I/O - Read files directly from HTTP to memory, no disk writes
• Range Request Support - Download only specific byte ranges from large files
• Smart In-Memory Cache - LRU cache with automatic eviction, no duplicate downloads
• Lightweight - WASM binary only ~500KB, runs anywhere
• Streaming Support - Pipe HTTP data directly to other commands
• Cross-Platform - Works in Node.js, Browser, and CLI
• No Dependencies - Minimal footprint, just HTTP and caching

Features

| Category | Features | |----------|----------| | Core | Virtual filesystem over HTTP, Zero-disk downloads, Range Request (bytes), In-memory LRU cache | | Performance | Streaming reads, Parallel downloads, Smart caching, Buffer pooling | | Platforms | Node.js (ESM + CommonJS), Browser (ESM), CLI Native, WASM | | CLI | Fetch files, Range download, Inspect metadata, Cache management, Pipe to commands | | Security | HTTPS support, CORS, No disk writes (memory only) | | Developer | TypeScript support, Simple API, Promise-based, Error handling |

Installation

From NPM

npm install http-ifs
npm install -g http-ifs

Requirements

Requirement Minimum Recommended Node.js 18.0.0 20.0.0+ RAM 128 MB 512 MB+ Storage 1 MB 10 MB+ OS Linux/macOS/Windows Ubuntu 22.04+

Quick Start

Node.js / TypeScript

import { HttpIFS } from 'http-ifs';

async function main() {
  const fs = new HttpIFS();
  
  // Download entire file from URL
  const data = await fs.read_file_all('https://example.com/file.bin');
  console.log('Downloaded:', data.length, 'bytes');
  
  // Download specific byte range (0-1024)
  const partial = await fs.read_file('https://example.com/file.bin', 0, 1024);
  console.log('Range size:', partial.length, 'bytes');
  
  // Cache statistics
  console.log('Cache:', fs.cache_stats());
  
  // Clear cache
  fs.clear_cache();
  
  await fs.close();
}

main();

Browser

<script type="module">
import init, { HttpIFS } from 'http-ifs';

await init();
const fs = new HttpIFS();

const data = await fs.read_file_all('https://example.com/file.bin');
console.log('Downloaded:', data.length, 'bytes');
</script>

CLI (Command Line)

# Download entire file
http-ifs fetch https://example.com/file.bin -o output.bin

# Download specific range (bytes 0-1024)
http-ifs fetch https://example.com/file.bin --range-start 0 --range-end 1024 -o partial.bin

# Inspect file metadata without downloading
http-ifs inspect https://example.com/file.bin

# Cache management
http-ifs cache stats
http-ifs cache clear
http-ifs cache warm https://example.com/file.bin

# Pipe to other commands
http-ifs stream https://example.com/file.bin --command "tar -xzv"
http-ifs stream https://example.com/data.txt --command "grep keyword"

CLI Usage

Basic Commands

# Fetch file
http-ifs fetch <url> -o <output-file>

# Fetch with range
http-ifs fetch <url> --range-start <start> --range-end <end> -o <output-file>

# Inspect file metadata
http-ifs inspect <url>

# Inspect with detailed headers
http-ifs inspect <url> --verbose

# Cache statistics
http-ifs cache stats

# Clear cache
http-ifs cache clear

# Warm cache (pre-download)
http-ifs cache warm <url>

# Stream to command
http-ifs stream <url> --command "<command>"

# Mount as filesystem (Linux with FUSE)
http-ifs mount <url> <mount-point>
http-ifs mount <url> <mount-point> --readonly

CLI Options

Command Description fetch Download file from HTTP URL inspect Get file metadata without downloading cache Manage memory cache stream Pipe HTTP data to other commands mount Mount URL as filesystem (FUSE)

Fetch Options

Option Description Default -o, --output Output file path stdout --range-start Start byte for range request 0 --range-end End byte for range request end of file -c, --chunk-size Chunk size for streaming 8192

API Reference

HttpIFS Class

const { HttpIFS } = require('http-ifs');
// or
import { HttpIFS } from 'http-ifs';

Constructor

const fs = new HttpIFS();

Methods

Method Description Returns read_file_all(url) Download entire file from URL Promise read_file(url, start, end) Download specific byte range Promise cache_stats() Get cache statistics string clear_cache() Clear all cached data void close() Close connection and cleanup Promise

read_file_all

const data = await fs.read_file_all('https://example.com/file.bin');

Parameter Type Description url string HTTP/HTTPS URL to fetch

Returns: Promise - File data as bytes

read_file

const data = await fs.read_file('https://example.com/file.bin', 0, 1024);

Parameter Type Description url string HTTP/HTTPS URL to fetch start number Start byte (inclusive) end number End byte (inclusive)

Returns: Promise - Range data as bytes

cache_stats

const stats = fs.cache_stats();
// Returns: "Size: 12345 bytes, Entries: 3"

Returns: string - Cache statistics

clear_cache

fs.clear_cache();

Clears all cached data from memory.

Binary Format

HTTP-IFS stores data in memory as raw bytes. The cache uses LRU (Least Recently Used) eviction policy.

Cache Structure

+----------+-----------+-----------+
| KEY      | VALUE     | TIMESTAMP |
| (string) | (Uint8Array) | (number) |
+----------+-----------+-----------+

Cache Eviction Policy

· Maximum size: 100MB (configurable) · LRU (Least Recently Used) eviction · Automatic cleanup when memory limit reached

Security

· HTTPS Support - Works with HTTPS URLs · CORS - Respects CORS policies in browser · No Disk Writes - Data never touches disk · Memory Only - All data stored in RAM · No Persistence - Cache cleared on restart

Usage Examples

Basic CRUD Operations

import { HttpIFS } from 'http-ifs';

async function example() {
  const fs = new HttpIFS();
  
  // Download file
  const data = await fs.read_file_all('https://example.com/data.bin');
  
  // Download partial
  const header = await fs.read_file('https://example.com/data.bin', 0, 100);
  
  // Cache stats
  console.log(fs.cache_stats());
  
  // Clear cache
  fs.clear_cache();
  
  await fs.close();
}

Download with Progress

import { HttpIFS } from 'http-ifs';

async function downloadWithProgress(url, outputFile) {
  const fs = new HttpIFS();
  
  console.log('Downloading:', url);
  const data = await fs.read_file_all(url);
  
  console.log('Downloaded:', data.length, 'bytes');
  console.log('Cache:', fs.cache_stats());
  
  // Save to file
  require('fs').writeFileSync(outputFile, data);
  
  await fs.close();
}

downloadWithProgress('https://example.com/file.bin', 'output.bin');

Stream to Command

import { spawn } from 'child_process';
import { HttpIFS } from 'http-ifs';

async function streamToCommand(url, command) {
  const fs = new HttpIFS();
  const data = await fs.read_file_all(url);
  
  const proc = spawn('sh', ['-c', command]);
  proc.stdin.write(data);
  proc.stdin.end();
  
  proc.stdout.on('data', (chunk) => {
    console.log(chunk.toString());
  });
  
  await fs.close();
}

streamToCommand('https://example.com/data.txt', 'grep keyword');

Using with Environment Variables

import { HttpIFS } from 'http-ifs';

async function example() {
  const fs = new HttpIFS();
  
  const url = process.env.FILE_URL || 'https://example.com/default.bin';
  const data = await fs.read_file_all(url);
  
  console.log('Data size:', data.length);
  
  await fs.close();
}

FAQ

Q1: What happens if the URL doesn't support Range Requests?

HTTP-IFS will attempt to download the entire file. If Range Requests are not supported, the read_file method will fall back to downloading the full file and then slicing the requested range.

Q2: How is the cache managed?

The cache uses LRU (Least Recently Used) eviction policy. When the cache reaches its limit (100MB default), the least recently accessed data is removed to make room for new data.

Q3: Is the data written to disk?

No. HTTP-IFS keeps all data in memory only. Nothing is written to disk unless you explicitly save it using fs.writeFile() or similar.

Q4: Can I use HTTP-IFS with very large files?

Yes. HTTP-IFS supports streaming and Range Requests, so you can download large files in chunks without loading the entire file into memory at once.

Q5: What's the performance impact?

HTTP-IFS is lightweight and uses in-memory caching. The main performance considerations are network latency and the size of the data being downloaded.

Q6: Can I use HTTP-IFS in a browser?

Yes. HTTP-IFS works in modern browsers with WASM support. Use the browser build from NPM or CDN.

Q7: Is HTTPS supported?

Yes. HTTP-IFS works with both HTTP and HTTPS URLs.

Q8: What happens if the server returns an error?

HTTP-IFS will return an empty Uint8Array and log an error message to console. You should handle errors appropriately in your code.

Q9: Can I customize the cache size?

Currently the cache size is fixed at 100MB. Future versions may support configurable cache size.

Q10: How do I contribute?

Fork the repository on GitHub, make your changes, and submit a pull request. All contributions are welcome.

Terms of Service

Please read these Terms of Service carefully before using HTTP-IFS.

1. Acceptance of Terms

By downloading, installing, or using HTTP-IFS (the "Software"), you agree to be bound by these Terms of Service.

2. Intended Use

HTTP-IFS is designed for legitimate purposes including:

· Reading files from HTTP/HTTPS URLs · Caching frequently accessed remote data · Streaming data from web to local applications · Prototyping and development · Edge computing and IoT applications

3. Prohibited Uses

You agree NOT to use HTTP-IFS for:

· Downloading illegal content · Bypassing security measures · Any activity that violates data protection laws · Building malware or harmful software · Exceeding rate limits of target servers

4. Responsibility and Liability

THE AUTHOR PROVIDES THIS SOFTWARE "AS IS" WITHOUT WARRANTIES. YOU ARE RESPONSIBLE FOR RESPECTING THE TERMS OF SERVICE OF TARGET SERVERS.

5. No Warranty

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

6. Limitations of Liability

IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DAMAGES ARISING FROM THE USE OF THIS SOFTWARE.

7. Ethical Reminder

Use this tool responsibly. Respect server resources, follow robots.txt rules, and don't abuse HTTP endpoints.

License

MIT License

Copyright (c) 2026 Dimzxzzx07

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.