I have been tasked with building a Node.js application that allows Users to download our complete data feed, which is approx. 50GB and is usually downloaded through curl, however they may test the calls in their browser (by specifying a limit). The data currently sits inside of a MySQL database, therefore I am using the node-mysql package. As soon as I make a request to the database and get 1 row, I write the data to the response.The Node application is running on a machine with 2 cores, 4GB RAM and 10GB of disk space. However, everytime I try to download the feed, the amount of Memory hikes up (monitored through pm2) and I get a curl: (18) error (A file transfer was shorter or larger than expected.). I've tried setting the header for Chunked Transfer Encoding.Can anyone share any light as to why I'm facing this problem? I'm using Node with Express to easily fulfil requests through the browser.Thanks in advance.
Submitted November 19, 2015 at 09:47AM by 01zombie01
No comments:
Post a Comment