I need to build an email extractor for a client. Basically users will be provided with a form where they can enter a google sheet url which contains the domain list. The scrapper will then scrape all the urls and extract emails from them. The list can vary anywhere 500 - 10,000 urls.I have already built this scrapping bot in Nodejs but the problem is - the bot sends requests to thousands of pages at once and due to low bandwidth (10 - 20 Mbps) the process is very very slow. I'm planning to rebuild this bot again using async/await as it helps in synchronous looping.How can I go about making this bot so that it can handle 100 - 500 users concurrently at once ? What would be the VPS specs suitable for this server ?
Submitted September 17, 2018 at 03:37PM by Ncell50
No comments:
Post a Comment