Tuesday 16 May 2017

stream consuming pace

Hi,I have a limited understanding of nodejs streams but I'm interested on how the data in each piped "consumer" flows. E.g:loadRemoteFile() .createReadStream() .pipe(progressReportTransformer) .pipe(unzipper) .pipe(someOtherTransformer) Imagine the upper situation where someOtherTransformer might "block" the stream by not calling done asynchronous but waiting for an async operation like creating a database table. It can work because a stream.Transformer can delay the data flow by not calling done() right away.Now is it a problem for the progressReporter, meaning will the progressReporter stream delay reading the next chunk until the one later (someOtherTransformer) in the "chain" finishes it's job?I don't know if I should imagine this process as a synchronous operation where the first pipe in the chain reads data from the readStream and when finishes pushes it/something to the next pipe for processing and this first pipe will only read the next chunk when the last pipe in the chain finished OR I should imagine it as if an intermediate pipe in the chain finished it's operation and pushed it's state to the next pipe for processing it gets ready for another chunk?I don't want my progressReporter to read (and report) faster than my latter async operation finishes it's operation.I hope I was clear enough.Thanks

Submitted May 16, 2017 at 03:41PM by Jim-Y

No comments:

Post a Comment