A project I'm working on grabs product data (in 100-product chunks) from an external API. I then need to create a CSV of the product data. There are upwards of 80,000 products. Any advice on how not to use gobs of memory? I've considered using Streams API but don't know if I'm barking up the wrong tree. Currently the application fails based on memory constraints.Current overview// Grab list of item ids // Batch ids in 100s // Throttled concurrent requests using Bluebird.js (5 at a time) // Usually fails at this point, too much memory consumed storing all the responses // Once all product data is retrieved, convert [response.data](https://response.data) to CSV // ftp upload csv to remote server Any advice would be much appreciated, I've never worked with this much data so optimization hadn't become an issue, and my current set up is obviously not scaleable. It runs locally, albeit slow, but I'm sure there are better ways to do this.
Submitted March 19, 2019 at 07:24PM by jfirwin
No comments:
Post a Comment