Node.js Streams Beyond the Basics: Memory, Backpressure, and Real Failures

Plenty of introductions to Node.js streams do a solid job with the basics: readable, writable, duplex, transform, pipe(), backpressure, a gzip demo, and sometimes a video range-request server as a grand finale. If you've already worked through one or two of those, you understand what a stream is and why pulling a 2GB file entirely into memory during a live demo is asking for trouble.

What those introductions rarely get into is what comes next. How much memory do you actually save by switching to streams, with concrete figures rather than a vague "less"? How do Node's native streams interact with the Web Streams API used by fetch and browsers? How do you stream output from a model running on your own hardware instead of paying per token through a hosted API? And which two or three mistakes eventually break a pipeline that looked fine in testing once it meets real traffic?

This article covers exactly that. It assumes you already know what a readable stream is. If that's not the case, the short recap below will get you up to speed quickly. Everything that follows builds on top of the fundamentals.

The 60-second recap

Readable   -> produces data              (fs.createReadStream, HTTP request body)
Writable   -> consumes data               (fs.createWriteStream, HTTP response)
Duplex     -> both, independently         (a TCP socket)
Transform  -> consumes, changes, produces (zlib.createGzip)

Readable stream -> Transform stream -> Writable stream

Data flows through the system in small pieces rather than as one big blob. pipe() wires streams together and handles the situation where the destination can't keep up with the source — that's backpressure. pipeline() accomplishes the same connection, but it also propagates errors from any stream in the chain and guarantees cleanup when something fails, which is why it's the function you want in any long-running service rather than a quick one-off script.

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('./input.txt'),
  createGzip(),
  createWriteStream('./input.txt.gz')
);

If that example already makes complete sense to you, feel free to skip ahead. What follows is the material worth knowing before putting stream-based code into a production environment for the first time.

The number nobody puts in these articles

Most writing on streams claims the buffered approach "uses a lot of memory" while the streaming approach "uses much less," without ever putting a number next to either claim. To fix that, a 501MB plain-text file was generated, and both approaches were run on the same machine, back-to-back, twice each.

// buffered.js: read the whole file, then scan it
import { readFileSync } from 'node:fs';

const before = process.memoryUsage().rss;
const data = readFileSync('./big.txt', 'utf8');
const afterRead = process.memoryUsage().rss;

let count = 0;
for (let i = 0; i < data.length; i++) if (data[i] === 'q') count++;

console.log({
  rssBeforeMB: (before / 1024 / 1024).toFixed(1),
  rssAfterReadMB: (afterRead / 1024 / 1024).toFixed(1),
  count
});

// streamed.js: read in 64KB chunks, scan as it arrives
import { createReadStream } from 'node:fs';

let peakRss = 0;
const timer = setInterval(() => {
  peakRss = Math.max(peakRss, process.memoryUsage().rss);
}, 10);

let count = 0;
const stream = createReadStream('./big.txt', { encoding: 'utf8', highWaterMark: 64 * 1024 });
stream.on('data', (chunk) => {
  for (let i = 0; i < chunk.length; i++) if (chunk[i] === 'q') count++;
});
stream.on('end', () => {
  clearInterval(timer);
  console.log({ peakRssMB: (peakRss / 1024 / 1024).toFixed(1), count });
});

Here's what actually came out of running these on a 500MB ASCII file with Node.js 22, two runs per approach:

Approach                         Peak RSS     Wall time    Notes
--------------------------------------------------------------------
readFileSync (buffered)          ~539 MB      1.24 - 1.76s  Full file materialized as a string
createReadStream, 64KB chunks    ~52 MB       1.56 - 1.67s  Roughly 10x less peak memory
createReadStream, 1MB chunks     ~50 MB       1.25 - 1.34s  Bigger chunks, same ballpark memory

A few caveats worth stating plainly, since leaving them out would make the results misleading: this reflects a single machine, a single file, and a single workload — a trivial character scan. Wall-clock time turned out roughly equal between the two approaches, with the streamed version even coming out slightly ahead in places, which was a bit unexpected. The assumption going in was that the overhead of processing many small chunks would slow things down more than it did. The memory numbers, on the other hand, matched expectations closely: the buffered run peaked at close to the full size of the file, while the streamed run peaked at a small figure that stayed largely flat regardless of file size. That contrast is really the entire case for using streams, condensed into one table — and it's easy to verify independently by running the two scripts above against a file of your own.

The practical implication: for a single request served by a sufficiently powerful machine, simply reading the whole file into memory is often good enough. The real issue was never a single request in isolation — it's the follow-up question that always comes next: what happens once you have 100 of those requests running simultaneously. Memory usage under the streaming approach barely shifts as concurrency grows, because each individual request only ever holds a chunk-sized buffer rather than the entire payload at once.

Node streams and Web Streams are the same family now

For years, Node.js streams and the WHATWG Streams API (ReadableStream, WritableStream, TransformStream — the classes behind fetch() in both the browser and Node) looked like two separate ecosystems with little overlap. That's no longer accurate. The node:stream/web module exposes the WHATWG classes directly, and Node's own Readable, Writable, and Duplex all come with .toWeb() and .fromWeb() methods, marked Stability 2 ("Stable") as of current Node.js releases.

This becomes relevant the moment you call fetch() from inside a Node service, since the response.body you get back is a Web ReadableStream, not a Node Readable.

import { createWriteStream } from 'node:fs';
import { Writable } from 'node:stream';
import { pipeline } from 'node:stream/promises';

const response = await fetch('https://example.com/large-report.csv');

if (!response.ok || !response.body) {
  throw new Error(`Download failed with status ${response.status}`);
}

// response.body is a Web ReadableStream, so convert the Node side instead
const fileStream = Writable.toWeb(createWriteStream('./report.csv'));

await response.body.pipeTo(fileStream);
console.log('Download saved without buffering the response in memory');

The conversion works just as smoothly in reverse — you can turn a Node Readable into a stream that any Web-stream-based API will accept:

import { createReadStream } from 'node:fs';
import { Readable } from 'node:stream';

const webStream = Readable.toWeb(createReadStream('./large-file.bin'));

// Now this works with fetch's request body, a browser upload API,
// or any library built against the WHATWG interface instead of Node's.
await fetch('https://uploads.example.com/put', {
  method: 'PUT',
  body: webStream,
  duplex: 'half' // required by fetch when the body is a stream
});

One detail that trips people up the first time: fetch will throw if you hand it a streaming request body without also passing duplex: 'half'. This exists because streaming request bodies aren't uniformly supported everywhere, so the spec requires you to opt in explicitly. It's a small thing that burns fifteen minutes once and then never again.

The real payoff is that your Node backend, your edge functions, and your browser code can now largely share the same streaming logic, rather than forcing you to maintain a fetch-flavored version and a separate pipe()-based version built around Node's fs and http modules.

stream.compose(): a pipeline that returns one stream

stream.compose() has technically been around since Node.js 16.9.0, but it spent a long stretch in experimental status before finally becoming stable in a recent Node.js release. Now that its API is settled, it deserves a closer look.

The idea: rather than authoring a full Transform subclass with _transform and _flush methods, you write an async generator function, and compose() collapses the entire chain of them into a single Duplex stream that you can pipe just like any other stream.

import { compose } from 'node:stream';
import { createReadStream, createWriteStream } from 'node:fs';

async function* toUpperCase(source) {
  for await (const chunk of source) {
    yield chunk.toString().toUpperCase();
  }
}

async function* onlyNonEmptyLines(source) {
  let leftover = '';
  for await (const chunk of source) {
    const lines = (leftover + chunk).split('\n');
    leftover = lines.pop() ?? '';
    for (const line of lines) {
      if (line.trim()) yield line + '\n';
    }
  }
  if (leftover.trim()) yield leftover;
}

const pipeline = compose(
  createReadStream('./raw.txt', { encoding: 'utf8' }),
  onlyNonEmptyLines,
  toUpperCase
);

pipeline.pipe(createWriteStream('./clean.txt'));

Set that beside writing two separate Transform classes — including manual buffering logic for the piece that splits lines — and you get the same underlying behavior with noticeably less scaffolding, plus code that reads top to bottom in the same order the data actually flows. compose() is also flexible about its endpoints: give it a plain AsyncIterable as the first argument and it becomes a readable Duplex; give it an AsyncFunction as the last argument and it becomes a writable sink. That means an entire pipeline can be built from nothing but async generators if that's the style you prefer.

None of this is a reason to rip out Transform classes that already work. But for new code — particularly anything that boils down to "parse this, filter that, reshape this" — defaulting to an async generator plus compose() tends to avoid writing several small Transform subclasses that all end up shaped identically.

Streaming tokens out of a local model

Nearly every "real-world streams" section in tutorials sticks to the same three examples: file downloads, gzip compression, and video range requests. Those are legitimate use cases, but there's a newer pattern that fits the streaming model just as well and rarely gets covered: pulling token-by-token output from a large language model.

Since a genuinely copy-pasteable example shouldn't depend on anyone's API key, the one below talks to Ollama running locally rather than a hosted provider:

# one-time setup
ollama pull llama3.2
ollama serve   # usually already running as a background service

By default, Ollama's /api/generate endpoint streams its reply as newline-delimited JSON (application/x-ndjson): one JSON object per line, each carrying a response fragment and a done flag, with the final line reporting "done": true. Strip away the formatting and this is just a readable stream doing light text parsing — the same chunk-based reasoning used everywhere else in this article carries over without modification.

// server.js: proxies Ollama's token stream to the browser over SSE
import { createServer } from 'node:http';

const server = createServer(async (request, response) => {
  if (request.url !== '/ask') {
    response.statusCode = 404;
    response.end('Not found');
    return;
  }

  response.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    Connection: 'keep-alive'
  });

  const ollamaResponse = await fetch('http://localhost:11434/api/generate', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'llama3.2',
      prompt: 'Explain backpressure in one short paragraph.',
      stream: true
    })
  });

  if (!ollamaResponse.body) {
    response.end();
    return;
  }

  // Web ReadableStream from fetch -> async iterate it directly (Node 18+)
  let buffered = '';
  for await (const chunk of ollamaResponse.body) {
    buffered += Buffer.from(chunk).toString('utf8');
    const lines = buffered.split('\n');
    buffered = lines.pop() ?? '';

    for (const line of lines) {
      if (!line.trim()) continue;
      const event = JSON.parse(line);
      response.write(`data: ${JSON.stringify({ text: event.response })}\n\n`);
      if (event.done) {
        response.write('event: done\ndata: {}\n\n');
        response.end();
        return;
      }
    }
  }

  // client disconnected mid-stream? stop pulling tokens we can't deliver
  request.on('close', () => {
    if (!response.writableEnded) response.end();
  });
});

server.listen(3000, () => console.log('Ask at http://localhost:3000/ask'));

If you want to inspect a raw NDJSON line from Ollama without the SSE framing wrapped around it, it looks like this:

{"model":"llama3.2","created_at":"2026-07-30T10:15:24.121Z","response":"Back","done":false}
{"model":"llama3.2","created_at":"2026-07-30T10:15:24.132Z","response":"pressure","done":false}
{"model":"llama3.2","created_at":"2026-07-30T10:15:24.166Z","response":" happens...","done":true,"done_reason":"stop"}

Two details here trip people up the first time around. First, in current Node.js versions, fetch's response.body is already async-iterable on its own, so there's no need to call Readable.fromWeb() just for this case — though that helper is still available if some downstream library expects a classic Node Readable object instead. Second, NDJSON lines can span multiple chunk boundaries, which is the exact same hazard that shows up when parsing CSV in streaming form: a single fetch chunk is not guaranteed to line up with a single JSON line. That's precisely why the example above buffers partial lines rather than calling JSON.parse(chunk) directly on whatever arrives.

If you'd rather hit a hosted API — OpenAI, Anthropic, or similar — the overall shape barely changes: you're still consuming a chunked HTTP response and forwarding it to the client as SSE. The local Ollama version just happens to be the one you can run immediately with no account or key required, and it's a reasonable choice for building out the client side of a feature like this during development, before pointing the same code at a hosted model once you need production-grade output quality.

Streaming a file straight into object storage

Sending a file "to S3" almost never involves a file small enough to fit comfortably in memory beforehand. The @aws-sdk/lib-storage package exists precisely because the plain PutObjectCommand requires knowing the content length before the request starts. Its Upload helper takes a stream instead, decides on its own whether a single request or a multipart upload is appropriate, and surfaces progress events along the way.

import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';
import { createReadStream } from 'node:fs';

const s3 = new S3Client({ region: 'us-east-1' });

async function uploadFile(localPath, bucket, key) {
  const upload = new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: key,
      Body: createReadStream(localPath)
    }
  });

  upload.on('httpUploadProgress', (progress) => {
    console.log(`${key}: ${progress.loaded} / ${progress.total ?? '?'} bytes`);
  });

  await upload.done();
  console.log(`${key} uploaded`);
}

To experiment without needing an actual AWS account, MinIO gives you an S3-compatible object store you can spin up locally with a single command. The exact same @aws-sdk/lib-storage code from above works against it with no changes beyond pointing the client at MinIO's endpoint rather than AWS:

docker run -p 9000:9000 -p 9001:9001 \
  -e "MINIO_ROOT_USER=admin" \
  -e "MINIO_ROOT_PASSWORD=admin12345" \
  minio/minio server /data --console-address ":9001"

const s3 = new S3Client({
  region: 'us-east-1',
  endpoint: 'http://localhost:9000',
  forcePathStyle: true, // required for MinIO's path-style URLs
  credentials: { accessKeyId: 'admin', secretAccessKey: 'admin12345' }
});

This is the approach worth defaulting to for anything larger than a few tens of megabytes: avoid resolving the entire file into a Buffer before passing it to the SDK. Feed in a createReadStream() and let Upload handle the rest — memory usage stays flat regardless of whether the file is 10MB or 10GB.

Parsing an upload without buffering the request body

The flip side of file uploads is the server that has to receive them, and this is where accidental buffering shows up most often — typically because a body-parsing middleware wasn't built with streaming in mind and silently read the whole multipart request into memory before your handler got a chance to touch it.

busboy handles multipart form parsing incrementally: as soon as bytes for a given file field start coming in, it hands you back a readable stream for that field instead of waiting until the request has arrived in full:

import { createServer } from 'node:http';
import { createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import busboy from 'busboy';

const server = createServer((request, response) => {
  if (request.method !== 'POST' || request.url !== '/upload') {
    response.statusCode = 404;
    response.end();
    return;
  }

  const bb = busboy({ headers: request.headers, limits: { fileSize: 200 * 1024 * 1024 } });
  const uploads = [];

  bb.on('file', (fieldName, fileStream, info) => {
    const destinationPath = `./uploads/${Date.now()}-${info.filename}`;
    uploads.push(
      pipeline(fileStream, createWriteStream(destinationPath)).catch((error) => {
        fileStream.resume(); // drain on failure so the request can still finish
        throw error;
      })
    );
  });

  bb.on('close', async () => {
    try {
      await Promise.all(uploads);
      response.writeHead(200, { 'Content-Type': 'application/json' });
      response.end(JSON.stringify({ status: 'ok' }));
    } catch (error) {
      response.statusCode = 500;
      response.end(JSON.stringify({ status: 'error', message: error.message }));
    }
  });

  request.pipe(bb);
});

server.listen(3000);

Nothing in this handler ever holds the full request body at once. Bytes move from the socket, through busboy's parser, and straight to disk. That distinction is what separates a service that copes fine with several large uploads happening in parallel from one that collapses the first time two users upload a video simultaneously.

Testing a custom transform stream

Hand-rolled Transform streams are prime candidates for quiet decay: nobody bothers writing tests for them, everyone assumes they "just work," and then a _flush edge case ships broken and nobody notices until production. Testing one is less painful than it sounds, though. You don't need to spin up a real file or a real HTTP server. Readable.from() combined with stream/promises gets you there.

// uppercase-transform.js
import { Transform } from 'node:stream';

export function createUppercaseTransform() {
  return new Transform({
    transform(chunk, encoding, callback) {
      callback(null, chunk.toString().toUpperCase());
    }
  });
}

// uppercase-transform.test.js
import { describe, it, expect } from 'vitest'; // works the same with Jest
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { createUppercaseTransform } from './uppercase-transform.js';

async function collect(readable) {
  const parts = [];
  for await (const chunk of readable) parts.push(chunk.toString());
  return parts.join('');
}

describe('uppercase transform', () => {
  it('uppercases each chunk', async () => {
    const source = Readable.from(['hello ', 'world']);
    const transform = createUppercaseTransform();
    source.pipe(transform);

    expect(await collect(transform)).toBe('HELLO WORLD');
  });

  it('propagates errors from upstream instead of hanging', async () => {
    const source = new Readable({
      read() {
        this.destroy(new Error('upstream failed'));
      }
    });
    const transform = createUppercaseTransform();

    await expect(pipeline(source, transform, async function drain(it) {
      for await (const _ of it) { /* discard */ }
    })).rejects.toThrow('upstream failed');
  });
});

The second test in that file matters more than it seems at first glance. It confirms that an error thrown upstream actually propagates outward, instead of leaving the transform stream stuck halfway through its work with a promise that never settles. That silent hang is the most common way a pipeline that looks like it's working turns into a stuck request once it's live.

A lightweight metrics tap

You don't need a full observability platform just to check whether a pipeline is behaving. Often a small PassThrough stream that tallies bytes and chunks as they flow past is all you need, and it drops into any existing pipeline without disturbing anything else:

import { PassThrough } from 'node:stream';

export function createMetricsTap(label) {
  let bytes = 0;
  let chunks = 0;
  const startedAt = process.hrtime.bigint();

  const tap = new PassThrough();
  tap.on('data', (chunk) => {
    bytes += chunk.length;
    chunks += 1;
  });
  tap.on('end', () => {
    const seconds = Number(process.hrtime.bigint() - startedAt) / 1e9;
    console.log(`[${label}] ${chunks} chunks, ${(bytes / 1024 / 1024).toFixed(2)} MB, ` +
      `${(bytes / 1024 / 1024 / seconds).toFixed(2)} MB/s`);
  });

  return tap;
}

await pipeline(
  createReadStream('./large-file.bin'),
  createMetricsTap('upload'),
  createGzip(),
  createWriteStream('./large-file.bin.gz')
);

If you want something more than console.log output, replace the logic inside the 'data' and 'end' handlers with a prom-client Counter or Histogram. The tap itself stays the same shape: it's just another stream sitting in the middle of a pipeline, reporting on what passes through.

Three mistakes that only show up under load

Unhandled 'error' events bring down the whole process. A stream that emits an error with no 'error' listener attached doesn't just fail its own request. In Node.js, an 'error' event on an EventEmitter that nobody handles gets thrown, and if nothing catches that throw, the process exits. It only takes one malformed upload to take an entire service offline because a manually wired pipe() chain had error listeners on two out of three streams. pipeline() exists specifically to make this class of bug structurally difficult, so prefer it, or make sure every stream you build by hand gets its own .on('error', ...), with no exceptions.

Reusing a stream instance leaks listeners. Say you wrap stream creation inside a function that gets called on every request, but something outside that function keeps a reference around and attaches its own listeners "just in case." Eventually you'll see MaxListenersExceededWarning show up in your logs, which is Node's way of flagging a memory leak in progress. Each request needs its own freshly created streams, and nothing outside that request's lifetime should be holding a reference to them once it's done.

Nobody destroys the stream when the client walks away. This is the mistake that actually costs money in production. If a user closes their browser tab mid-download and you're not listening for response.on('close', ...) to call .destroy() on the source stream, the server carries on pulling data from disk, or, in a worse case, leaves a database cursor open or keeps an upstream HTTP connection alive, all for a client that has already left. Under enough concurrent load, that's exactly how a server quietly exhausts its file descriptors or connection pool with no obvious cause in the logs. The fix is always the same three lines: listen for the destination to close, destroy the source in response, and treat that pattern as a default habit on every stream-based route rather than something you remember after an incident.

When to reach for which API

pipe()
  Simple, two-or-three-stream chains. Handles backpressure.
  Error handling is your responsibility per-stream.

pipeline()
  Anything running in production. Handles backpressure,
  forwards errors from every stream, cleans up on failure.
  Has a promise-based version and supports AbortSignal.

stream.compose()
  New pipelines built from async generators instead of
  Transform subclasses. Returns one Duplex you can pipe
  like any other stream. Stable as of recent Node.js releases.

node:stream/web (ReadableStream/WritableStream/TransformStream)
  Anywhere you're mixing Node streams with fetch(), browser
  code, or a library built against the WHATWG standard.
  Convert with .toWeb() / .fromWeb() at the boundary.

None of these approaches makes the others unnecessary. They each solve for a different piece of the same overall flow of data through your system. It's common now for a single pipeline to begin with a Web ReadableStream coming from fetch, convert that data, feed it through a transform assembled with compose(), and finally pass the output to pipeline() to handle the write. Mixing APIs this way used to feel unusual; today it's a fairly ordinary pattern.

Closing thoughts

Nothing here removes the need to first understand backpressure, highWaterMark, and the four fundamental stream classes. That base knowledge remains a hard requirement before any of the techniques above are useful. What does change once you have that background is where the real engineering work begins: confirming memory improvements with a measured figure rather than a loose claim, remembering that a fetch-based stream and a Node stream only need one .toWeb() conversion to interoperate, and building in a plan for a client disconnecting mid-request as a standard part of writing the route, not a detail you patch in afterward.

If you run the benchmark examples mentioned earlier on different hardware and the results diverge noticeably from what's shown here, that's a useful data point worth sharing rather than something to brush aside, because numbers around throughput and memory usage hold up much better once they've been checked against more than one machine.

Resources

• Node.js documentation covering the classic Stream API: https://nodejs.org/api/stream.html
• Node.js documentation covering the Web Streams API: https://nodejs.org/api/webstreams.html
• The Ollama reference for its streaming endpoint: https://docs.ollama.com/api/streaming
• The @aws-sdk/lib-storage package listing on npm: https://www.npmjs.com/package/@aws-sdk/lib-storage
• The busboy project repository: https://github.com/mscdex/busboy
• The MinIO project site: https://min.io/