fs.writeFile vs SQL insert performance.

I'm wondering what would be the best way of handling large quantities of data. Currently i'm working on a node program that takes read stream data and logs it for testing purposes, the stream sends new data up to multiple times per second and the current way i'm logging it is by adding each new chunk into an array and then using fs.writeFile() to save it to a JSON file. Now this works fine, but starts to lag and fail once the array length gets into the 10s of thousands. Also mind that the program is doing this for a little over 200 streams at the same time.Using a write stream isn't an option as i need all the data formed into a single array of objects, not just a list of seperate objects. So my question is which would be a better approach performance wise when using it between 200-1000 times per second, using my current approach of adding each tick to an array then using fs.writeFile() or using SQL queries?

Submitted April 03, 2018 at 09:34PM by captainXcannabis