Wednesday 22 August 2018

How to download large files efficiently?

Hello there :)For my desktop app using NodeJS, our users can download content (multiple dozen megabytes each) to use the app with.At first I went with FTP downloads and it worked well for me and some users, but others had issues with downloads stalling randomly, and not even throwing errors, even, just hanging the download process.I switched to HTTP by using got or download. Again, it worked for some users, but others have had similar issues with FTP.As a last resort, I've made a system call to curl to download the files I wanted, and again, for many users it solved things, but lately I've had a few reports of curl not even starting up, and the download hanging there (somehow, Windows won't let my node app start up curl, even though if the user starts curl separately, it works fine, meh.)As a last, last resort, I offered a rsync command to my users so they could download their content, and it worked flawlessly every single time, for everyone.My question is : does anyone have had similar issues ? Or has a recommendation on which module I should use to make these downloads safe?Something with automatic retrying, and events to follow on download progress ?I've already rewritten my file download functions a dozen times already, I'm a little fed up with that and would like to find a solution to my (users') issues for good.Any idea?

Submitted August 22, 2018 at 04:01PM by AxelTerizaki

No comments:

Post a Comment