I created a Node.js bot to easily scrape those pages protected by JavaScript challenge - like CloudFlare's anti DDoS protection.If you're not using a headless browser like Selenium (Which is a huge overkill for scraping tbh) those challenges are impossible to bypass with regular frameworks (axios, request, etc) and the site can't be accessed.My bot parses and solves them - and presents the HTML of the original protected site.You can check it out here - https://ift.tt/2BsrxUT hope I'll manage to find the time to expand it to handle other anti-bot mechanisms/techniques. Any tips and suggestions are appreciated :)
Submitted November 23, 2018 at 01:11PM by evya135
No comments:
Post a Comment