I'm building a very simple web crawler with node 7 and have come across some strange issues. I've fairly new to node/async type things but this seems strange.I have a timer that gets a job from beanstalk every 20ms (to throttle my requests to ~ 50/sec to sites) .. I have the function keep track of the time each time it runs. When i just do request() i see consistent timestamps where req #50 is at the 1s mark and #100 is very close to the 2 second mark etc.. But when I introduce cheerio into the mix after the request, suddenly my timer slows way the hell down. How could cheerio screw up a function that calls itself every 20ms?
Submitted April 18, 2017 at 02:54PM by hecktarzuli
No comments:
Post a Comment