Why Decoding Buffer Chunks as Text Breaks File Uploads

A file upload endpoint might pass every manual test you throw at it. Small images, PDFs, plain text files, everything goes through cleanly. Then, without warning, a customer uploads a file that comes back corrupted, an image with a scattering of wrong pixels, or a payload that fails to parse as JSON even though it was valid before it left the client. No one touched the file in transit. The damage happened quietly, inside code that looks entirely sensible at first glance, and the root cause is one of the most frequent slip-ups in Node.js: treating binary data as though it were text.

What a Buffer Actually Is

A Buffer is simply Node's way of holding a sequence of raw bytes in memory. It carries no built-in meaning and no character encoding, just plain numeric values between 0 and 255 stored contiguously:

const buf = Buffer.from([72, 101, 108, 108, 111]);
console.log(buf); // <Buffer 48 65 6c 6c 6f>
console.log(buf.toString("utf8")); // "Hello"

Those five byte values only turn into the readable string "Hello" once you deliberately interpret them using a particular encoding, UTF-8 in this example. On their own, the bytes are not text. They're just bytes, and a Buffer exists precisely to let you work with binary data before, or entirely without, deciding that it should be read as characters. That gap between "raw bytes" and "text under a chosen encoding" is exactly where this whole class of bug comes from.

The Mistake: Decoding Binary Data as Text

The pattern that triggers this problem is deceptively ordinary-looking:

app.post("/upload", (req, res) => {
  let body = "";
  req.on("data", (chunk) => {
    body += chunk.toString("utf8"); // corrupting the file, one chunk at a time
  });
  req.on("end", () => {
    fs.writeFileSync("upload.png", body, "utf8"); // and corrupting it again here
  });
});

Image bytes are not text. They form an arbitrary binary stream encoding pixel values, compression tables, and metadata, none of which was ever intended to be read as UTF-8 characters. Calling .toString("utf8") on that binary payload forces the runtime to interpret bytes that frequently don't correspond to any valid UTF-8 sequence. Rather than throwing an error, the decoder quietly swaps in the Unicode replacement character (�, U+FFFD) wherever it hits a byte pattern it can't decode. Those original bytes are gone for good, replaced by a placeholder with no way back to the original value. This is exactly why the resulting corruption seems scattered and random: only the byte sequences that don't happen to be valid UTF-8 get mangled, and for binary formats like images, that happens constantly.

app.post("/upload", (req, res) => {
  const chunks = [];
  req.on("data", (chunk) => chunks.push(chunk)); // keep raw bytes, don't decode anything
  req.on("end", () => {
    const fileBuffer = Buffer.concat(chunks);
    fs.writeFileSync("upload.png", fileBuffer); // write raw bytes, no string conversion involved
  });
});

Assuming the request body carries the raw file bytes directly rather than a multipart/form-data payload, this approach preserves the file exactly as uploaded. If you're dealing with multipart uploads, run the data through a proper multipart parser first to extract the file portion. The underlying fix is straightforward: never convert binary data into a string at all. Instead, gather the incoming Buffer chunks as-is, join them at the byte level, and write those bytes straight to disk or storage.

The Same Bug, Smaller and Sneakier: Multi-Byte Characters Split Across Chunks

Even when you're genuinely working with text, decoding chunk by chunk rather than all at once opens the door to a related but subtler flavor of this bug, particularly relevant if you're processing streamed data incrementally:

readStream.on("data", (chunk) => {
  process.stdout.write(chunk.toString("utf8")); // can corrupt multi-byte characters
});

A single emoji or accented letter can span several bytes under UTF-8, and chunk boundaries from a network connection or file stream have no idea where those multi-byte boundaries fall. If a chunk happens to end mid-character, decoding that chunk in isolation yields a broken character, silently swapped for a replacement character, even though the full, correct byte sequence was present all along, just split across two separate .toString() calls, each of which only saw half of it.

const decoder = new (require("string_decoder").StringDecoder)("utf8");

readStream.on("data", (chunk) => {
  process.stdout.write(decoder.write(chunk)); // holds incomplete multi-byte sequences until the rest arrives
});

readStream.on("end", () => {
  process.stdout.write(decoder.end());
});

Node's built-in StringDecoder is designed exactly for this situation: it withholds any incomplete multi-byte sequence sitting at the tail end of a chunk instead of decoding it too early, and waits for the remaining bytes to show up in the next chunk before finishing the character. This is a distinct fix from concatenating buffers with Buffer.concat, one that applies specifically when you need to decode text safely as it streams in, rather than buffering an entire binary file before doing any conversion at all.

Encoding Mismatches: Writing One Encoding, Reading Another

There's a related failure that's just as quiet: choosing mismatched encodings on the writing side versus the reading side of an operation.

const token = crypto.randomBytes(32); // raw binary
const encoded = token.toString("base64"); // encode once, deliberately, for safe transport

// later, elsewhere in the codebase
const decoded = Buffer.from(encoded, "hex"); // wrong encoding — does not recover the original bytes

Buffer.from reads the string according to whatever encoding argument you pass it. If that encoding isn't the one actually used to produce the string in the first place, you won't get back the original bytes in any dependable way. Depending on which encoding is involved and what the input looks like, Node might decode entirely different byte values, or silently drop the parts of the string that don't fit that encoding's rules, rather than raising an error you'd notice immediately.

Buffer.alloc vs Buffer.allocUnsafe: A Security-Relevant Difference, Not Just a Performance One

There's one more distinction worth internalizing, and it matters here specifically because getting it wrong isn't just a bug, it's a potential leak of sensitive data:

const safeBuf = Buffer.alloc(16);       // zero-filled, always
const fastBuf = Buffer.allocUnsafe(16); // NOT zero-filled — may contain old memory contents

Buffer.allocUnsafe skips the step of zeroing out the memory it hands you, which is genuinely faster, but it means the buffer can still hold whatever bytes happened to be sitting in that region of memory beforehand, possibly a leftover piece of an earlier request, a fragment of some other user's session token, or anything else that was previously stored there. If you allocate a buffer this way and only write part of it before sending it out, whether over a network connection or into a file on disk, you risk exposing data that had nothing to do with the current operation. Buffer.alloc pays a small, predictable cost to zero the memory up front, and that should be your default. Reach for allocUnsafe only in the narrow case where you're certain you'll overwrite the entire buffer yourself before anything else touches it.

The Actual Lesson

Every one of these failures comes back to a single underlying confusion: treating a raw sequence of bytes as though it were naturally text, when in reality "text" only exists once you've made a deliberate decision about which encoding to use to interpret those bytes, and that decision can be applied incorrectly, too soon, or at the wrong point in the pipeline. The safer approach is to keep binary data binary for as long as possible, concatenating and transforming it as raw bytes, and to convert it into a string only at the specific moment something genuinely needs it as text, using the correct, explicitly chosen encoding at that point. Without that discipline, corruption doesn't show up as an obvious error. It just quietly substitutes whatever bytes it couldn't interpret, and you end up discovering the problem later, usually because a user reported that something didn't work.