Node.js Streams v2: Navigating Backpressure and Readable/Writable Modes

The Buffer Overflow Threat (Fast Source, Slow Destination)

In early Node.js streams, readable streams pushed data as fast as they read it.

• ◆The Issue: If a readable stream reads a file at 100MB/s, but the writable destination (like a slow HTTP network connection) can only write at 5MB/s, the system must buffer the data in RAM.
• ◆The Result: Memory usage spikes, causing server instability.

This mismatch is resolved by the Streams v2 API and Backpressure management.

Stream Principle: A writable stream must signal the readable source to pause when its internal buffers are full, resuming once empty.

The Stream v2 Pull-Based Model

Streams v2 shifts from push-based streams to a pull-based model using the readable event:

// Managing backpressure and stream data events in Node.js
const fs = require('fs');
const readable = fs.createReadStream('large-dataset.csv');
const writable = fs.createWriteStream('destination.csv');

readable.on('readable', () => {
    let chunk;
    // Pull data only as long as writable stream can consume it
    while (null !== (chunk = readable.read())) {
        const canWrite = writable.write(chunk);
        if (!canWrite) {
            // Buffer is full. Pause reading to prevent overflow.
            readable.pause();
            break;
        }
    }
});

writable.on('drain', () => {
    // Buffer is empty. Resume stream reading.
    readable.resume();
});

By handling backpressure, Node.js applications process files of any size without risking memory spikes.