Node.js Streams Explained: Fixing Out-of-Memory File Crashes

Picture a production server that goes down in the middle of an ordinary, quiet afternoon.

There was no traffic spike and no flood of concurrent users. Just a single person using the application, who clicked a button to export a large report.

Within seconds, the process stopped responding entirely, and the console printed a familiar message: "JavaScript heap out of memory."

If you've run into that error before, you know how unsettling it is.

The natural reaction is confusion. How could one file, requested by one user, take down a whole running application?

That kind of incident is a great teacher. It points straight at a core Node.js concept that every backend developer eventually needs to understand: streams.

The Big Mistake Most Beginners Make

When developers are new to Node.js, they usually reach for the simplest tools available.

For reading a file from disk, the go-to choice is often fs.readFile(). It's approachable: you pass in a path, use a callback or await, and the full contents of the file come back to you.

A typical version of this looks like:

import fs from 'node:fs/promises';

async function sendFile(filePath) {
  // Reading the entire file at once
  const bigData = await fs.readFile(filePath);
  return bigData;
}

This approach performs fine as long as the files stay small. A 50-kilobyte text file loads instantly. A small profile picture is no issue either.

Since everything checks out during local testing, it's tempting to assume the code is production-ready as-is.

Then reality sets in.

Why Reading Everything at Once Fails

Consider how your machine's RAM actually gets used. When fs.readFile() runs, Node.js pulls the whole file into memory, byte by byte, before handing it back to you.

Suppose your server only has 1 gigabyte of RAM allocated to the app.

Now suppose a user tries to upload a video, or requests a 900-megabyte raw log file.

Calling fs.readFile() on that 900-megabyte file triggers a chain reaction:

• Node.js immediately requests 900 megabytes of memory from the operating system.
• The garbage collector works overtime as available memory shrinks.
• If a second user requests the same file simultaneously, memory demand jumps to 1800 megabytes.
• The server exhausts its memory budget and crashes outright.

The failure isn't caused by a corrupted file. It happens because the whole payload is being swallowed in a single gulp instead of being consumed gradually.

What Are Streams in Simple Words?

Step away from code for a second and think about a real-world analogy.

Suppose you need to move water from a large lake into your backyard garden.

You wouldn't try to scoop the entire lake into one enormous bucket and carry it over — that's simply too much weight for anyone to lift.

Instead, you'd hook up a garden hose.

Water flows through that hose in a thin, continuous stream: a bit enters at one end, travels along the pipe, and exits at the other end onto the soil.

With nothing more than a narrow hose, you can transport millions of liters over time, never once lifting the full volume at once.

A stream in Node.js works exactly like that hose.

Rather than pulling an entire file into memory in one shot, a stream reads it in small, digestible pieces known as chunks.

By default, a chunk is usually around 64 kilobytes.

Node.js grabs one chunk, processes it, forwards it to wherever it needs to go, and then releases it from memory before moving on to the next chunk.

This is why a server can push a 10-gigabyte file through while consuming only about 20 to 30 megabytes of RAM.

The Four Types of Streams in Node.js

Node.js exposes four fundamental building blocks for working with streaming data. You don't need to master every detail immediately, but it's worth knowing what each one is called:

1. Readable Streams

A readable stream is one you pull data from.

• Examples include reading a file from disk, receiving the body of an incoming HTTP request, or reading rows back from a database query.

2. Writable Streams

A writable stream is one you push data into.

• Examples include writing content to a new file, sending a response back to a browser, or writing bytes out over a network socket.

3. Duplex Streams

A duplex stream lets you do both jobs at once: you can read from it and write to it simultaneously.

• Example: a network connection, such as a TCP socket, where you're sending data out and receiving data back over the same connection.

4. Transform Streams

A transform stream is a specialized duplex stream. Its job is to change the data as it passes through, rather than just moving it along unchanged.

• Example: compressing a file into .gzip format as it flows through, or encrypting text on the way to disk.

Seeing the Difference: Code Examples

Let's compare these approaches with a concrete scenario. Imagine you're building a basic HTTP server that lets visitors download a large file.

The Bad Way (High Memory Usage)

JavaScript

import http from 'node:http';
import fs from 'node:fs/promises';

const server = http.createServer(async (req, res) => {
  try {
    // We load the whole file into RAM first
    const fileData = await fs.readFile('./massive-dataset.csv');

    res.writeHead(200, { 'Content-Type': 'text/csv' });
    res.end(fileData);
  } catch (error) {
    res.writeHead(500);
    res.end('Something broke');
  }
});server.listen(3000);

If massive-dataset.csv happens to be 2 gigabytes in size, this code will attempt to hold the entire 2 gigabytes in memory before sending even a single byte back to the client. On most cloud hosting setups, this will cause the process to crash right away.

The Better Way (Low Memory Usage)

Now let's build the same download feature using streams instead:

JavaScript

import http from 'node:http';
import fs from 'node:fs';

const server = http.createServer((req, res) => {
  // We create a readable stream
  const readStream = fs.createReadStream('./massive-dataset.csv');  res.writeHead(200, { 'Content-Type': 'text/csv' });  // We connect our read stream directly to the response
  readStream.pipe(res);  readStream.on('error', (err) => {
    res.writeHead(500);
    res.end('File not found or error reading');
  });
});server.listen(3000);

Notice the call to .pipe()?

That one method call accomplishes something powerful. It wires our file-reading stream directly into the outgoing HTTP response (res).

The moment the disk delivers the first small chunk (say, 64 KB), Node.js forwards it to the client immediately. There's no need to wait until the entire file has been read. Memory usage remains small and steady for the whole duration of the download.

Understanding Backpressure (The Traffic Jam Problem)

There's a key concept in streaming that every developer should grasp: backpressure.

Go back to the garden hose analogy for a moment.

Imagine pushing water into a pipe at 100 liters per second, while the outlet valve only lets 10 liters per second escape.

Pressure keeps building inside the pipe, and if the pipe isn't strong enough, it bursts.

The same kind of problem shows up in software all the time. An SSD can supply data at hundreds of megabytes per second. Meanwhile, the person downloading your file might be on a slow mobile connection.

So if Node.js keeps pulling data off the disk faster than the client can receive it, where does that surplus data end up?

It piles up in your server's RAM, waiting to be sent.

Left unchecked, this defeats the whole point of using streams, since your memory consumption climbs right back up.

How Modern Node.js Solves This

Fortunately, current versions of Node.js include a built-in solution for exactly this issue: the pipeline function, available from the stream/promises module.

Rather than relying on the older .pipe() approach, modern code should favor pipeline:

JavaScript

import http from 'node:http';
import fs from 'node:fs';
import { pipeline } from 'node:stream/promises';

const server = http.createServer(async (req, res) => {
  const readStream = fs.createReadStream('./massive-dataset.csv');  try {
    // pipeline handles backpressure and cleans up automatically
    await pipeline(readStream, res);
  } catch (error) {
    if (!res.headersSent) {
      res.writeHead(500);
      res.end('Transfer failed');
    }
  }
});server.listen(3000);

So what makes pipeline a better choice than .pipe()?

• It reacts to mismatched speeds: when the client is slow to receive data, it automatically pauses the readable stream until the client can accept more.
• It handles errors gracefully: if someone closes their browser mid-download, pipeline shuts down the read stream and releases the file handle properly, preventing memory leaks.

Real World Situations Where Streams Save You

Streams aren't reserved only for shipping huge video files or giant downloads. They quietly show up in all sorts of everyday production scenarios:

• Log Processing: Scanning a massive server log for errors doesn't require loading the entire file into memory. You can stream through it line by line instead.
• Image and Video Transformations: When someone uploads a high-resolution photo, you can pipe the incoming upload directly into an image-resizing tool, skipping the step of writing the raw file to disk first.
• Database Exports: When exporting millions of rows into a CSV, pull rows in small chunks from a database cursor and stream them straight to the client as they arrive.
• Data Encryption: Encrypting sensitive information on the fly as it's being written out to cloud storage.

Common Mistakes to Avoid

Even developers who understand the theory behind streams can trip over a few practical pitfalls:

• Skipping error handlers: Older stream APIs don't propagate errors automatically. If a step in your pipe throws and nothing is listening for it, the whole process can go down. Stick with pipeline, or explicitly listen for the 'error' event.
• Turning streams back into buffers: It's tempting to collect every 'data' event into an array and then concatenate everything into one big string or buffer. Doing this cancels out the memory benefits you were trying to get in the first place.
• Leaving resources open: If an operation fails partway through, make sure any open file descriptors get closed properly instead of lingering.

Final Thoughts

When developers are new to coding, they tend to picture data as something fixed and complete, sitting there waiting to be used, whether that's an entire file, a full database table, or a finished response.

Working professionally on backend systems requires letting go of that mental model.

Data isn't always a solid, unmovable object. More often than not, it behaves like a flowing river.

You don't need to scoop up an entire river to interact with it. You just need to let it flow past you, a little at a time.

Once streams become part of your toolkit, large files stop being something to dread. Your infrastructure can run on leaner, less expensive servers. Your applications feel more responsive to the people using them. And perhaps most importantly, you can rest easy knowing that an unexpectedly large 2GB upload won't take your server down in the middle of the night.