I'm taking my first shots at making a simple CSV parser using Node JS streams. After working on this since I got into work this morning, I've managed to produce the following:var transformer = transform(function(record, callback) { setTimeout(function() { var parsedRecord = parseRecord(record); callback(null, '"'+parsedRecord.join('","')+'"\n'); }, 500); }, {parallel: 10}); var files = fs.readdirSync(inDir); var count = -1; function processNext() { count++; if(count === files.length) { return; } console.log(files[count]); let inStream = fs.createReadStream(__dirname + inDir + files[count]); let outStream = fs.createWriteStream(__dirname + outDir + files[count]); // I was getting a "max listeners exceeded error", so I fudged it here. inStream.setMaxListeners(0); outStream.on('close', function() { setTimeout(processNext, 500); }); inStream .pipe(parser) .pipe(transformer) .pipe(outStream); } processNext(); I've stripped it of its wordiness as much as I thought possible.parser is an instance of csv-parse. Takes a CSV line in, spits out an array.parseRecord is my own parsing transform. Takes the array from parser in and spits out a string I want to go to outStream.The timeout on the 'close' event is there on a hunch, since transformer had the same delay. Figured maybe it was writing to the outStream after it was closed, but that doesn't make sense anymore.ProblemI'm still getting a write after end error immediately after the listener launches a second iteration of processNext. I thought making a new inStream for each iteration would get around that, since it works just fine with the first iteration.What am I not getting about streams?
Submitted April 06, 2017 at 11:49PM by pm_your_moneymaker
No comments:
Post a Comment