Streams are often introduced as a way to process files without loading everything into memory. That is true only when data producers respect consumers. If a fast source keeps writing while a slow destination falls behind, the application has simply moved the full dataset into an ever-growing memory buffer.
The boolean returned by write matters
A writable stream returns false when its internal buffer has reached the high-water mark. That is a request to pause. Resume after the drain event, when the destination has processed enough buffered data.
async function writeChunks(writable, chunks) {
for await (const chunk of chunks) {
if (!writable.write(chunk)) {
await once(writable, 'drain')
}
}
writable.end()
}
Ignoring the return value may appear fine in local testing because disks and loopback networks are fast. Under a slow client or remote object store, buffered data grows, garbage collection becomes expensive, and the process can be terminated for exceeding its memory limit.
Use pipeline for lifecycle management
stream.pipeline connects backpressure and also coordinates errors and cleanup across every stage. If compression fails, the file reader and HTTP response should not remain open. The promise-based API makes that lifecycle explicit.
await pipeline(
createReadStream(input),
createGzip(),
createWriteStream(output)
)
Transform streams must follow the same contract. Call the transform callback only when the transformed chunk is ready, and avoid starting unlimited asynchronous work inside _transform. If a transform performs remote calls, add an intentional concurrency limit.
Backpressure crosses service boundaries
The concept is larger than Node’s stream classes. A queue consumer, WebSocket broadcaster, CSV exporter, and database pagination loop all need a policy for a slow downstream dependency. That policy may be pausing, bounding concurrency, dropping replaceable updates, or spilling durable work to a queue.