If I need to spread an increasing large amount of tasks over a few servers, should I spin up a bunch of VPSes and add keep adding more? Or is there like a cloud scaling service Amazon offers or something?

I need to do a bunch of HTTP requests and then a bunch of processing of the resulting data. It gets quite intensive, to the point that I can't reliably do it on one server anymore, so I created two more Linode VPS in addition to the main one to spread the work load over.The work is basically the main server sends a bunch of commands to the other servers to do, which includes an HTTP request and what to do with the results, then the other servers perform the fetches, manipulate the data then store it in a central database.This is working now but the work load is increasing and I'll need to spin up another VPS soon, and the configuration is kind of annoying.Is there a better way to do this? An automated way to create more servers as needed that isn't super hard to understand? Creating new VPSes is easy just time consuming.

Submitted October 31, 2018 at 02:41PM by canucktesladude