The Buffered Memory Crisis
When processing large files (like 2GB database backups or high-resolution video streams), reading files into memory using fs.readFile will crash servers because the system runs out of RAM.
Node.js streams solve this by reading and writing files in small chunks (buffer blocks) sequentially, keeping memory consumption low.
Stream Principle: Keep memory usage constant regardless of file sizes. Process data in chunks as it flows from source to destination.
The Four Types of Streams
Node.js defines distinct stream types:
- ◆Readable: Sources of data (e.g.
fs.createReadStream). - ◆Writable: Destinations of data (e.g.
fs.createWriteStream). - ◆Duplex: Streams that are both Readable and Writable (e.g. TCP sockets).
- ◆Transform: Duplex streams that modify data as it is written (e.g. gzip compression).
| Stream Type | Event Triggers | Core Methods |
|---|---|---|
| Readable | data, end, error | read(), pipe(), pause(), resume() |
| Writable | drain, finish, error | write(), end(), cork() |
Implementing Streaming Pipelines with Piping
The .pipe() method connects streams, handling memory buffers automatically:
javascriptcode
// Compressing files via streams in Node.js
const fs = require('fs');
const zlib = require('zlib');
const source = fs.createReadStream('/data/users.csv');
const destination = fs.createWriteStream('/data/users.csv.gz');
const gzip = zlib.createGzip();
// Pipe data from source, compress it, and write it to disk
source.pipe(gzip).pipe(destination);
console.log("File compression pipeline active.");By avoiding buffering files in RAM, Node.js applications process high-volume files with stable server memory.
VP
Vijay Paliwal
Founder, SHIVAM ITCS · 18+ years enterprise & AI engineering
MCA · Ex-HiveGPT USA · Ex-Social27 Seattle