Backpressure problem in http.ServerResponse

Hi guys, I'm new to Node so I'm practicing some backend development using only pure Node (no frameworks) to get a feel for it, and I thought I'd give streams a try to process a somewhat large text file (a 200 MB .txt file).I think I have a backpressure problem in my code and I'm not sure how to solve it. Here's the code:// The default chunk size of Streams created by createReadStream is 64kb, which // is far too small for big files. This CHUNKSIZE is set to 1 MB const CHUNKSIZE = 1048576; const server = http.createServer((req, res) => { if(req.url == "/") { // The following hack doesn't seem to work //res.socket._writableState.highWaterMark = CHUNKSIZE; const stream = fs.createReadStream(__dirname + FILENAME, {highWaterMark: CHUNKSIZE}); // outputs "16384", as expected. If I uncomment the hack above, it does // show CHUNKSIZE, but still doesn't seem to work console.log("res.writableHighWaterMark: " + res.writableHighWaterMark); let fileChunkNum = 1; stream.on("data", (chunk) => { console.log("\nFile stream received chunk #" + fileChunkNum++); console.log("Megabytes read so far: " + (stream.bytesRead / 1024 / 1024) + " MB"); }); // FIXME: I guess this is taking forever because of res' default highWaterMark (16kb) stream.pipe(res); } else { res.end("Request not handled"); } }); Now, I've been reading about Streams and the problem here seems to be that there's no way to set the highWaterMark of an already created Stream (in this case, the http.ServerResponse that createServer passes as the second argument to my callback). I've tried to hack around it by manually setting the highWaterMark of the ServerResponse's socket's _writableState but that doesn't seem to work (even though the correct value is displayed when I console.log it). If I remove the call to stream.pipe, the chunks are very quickly logged in the console, so I guess what's happening is that res' paltry 16kb buffer is getting overwhelmed. Is there a solution I am not seeing? Is my whole approach just completely wrong? I'm open to any suggestions.

Submitted March 25, 2020 at 08:15PM by El_bigote_de_Dali