Friday, 7 September 2018

How do I authenticate with facebook to scrape profile pages without external libraries?

Is there a way I can wrap this request in a post that logs me into facebook and then does this get request to scrape profile links? I am blocked because I am not authenticated, otherwise I should be printing all the links on the requested page from facebook. Is there a way I can authenticate myself and use the cookie value to keep me logged in so I can scrape these links in FB? request.get('https://ift.tt/2MXHqtB', { options: { 'User-Agent': 'Mozilla/5.0' } }, (err, resp, body) => { if (!err) { // If the response is 200, then use cheerio to make a jquery object for interacting with the DOM and grab links const $ = cheerio.load(body); links = $('a'); //jquery get all hyperlinks var linkArray = []; // Array to store all valid Facebook results // Ensure each link is from Facebook and is at least searching for "Profiles". // This guarantees only facebook profile links in a later query // that uses these links for new search engine url queries, by specifying a domain and a page attribute keyword. $(links).each((i, link) => { // if ($(link).attr('href').includes('facebook.com') && $(link).text().includes('Profiles')) { // linkArray.push($(link).attr('href')); // } console.log(($(link).attr('href'))); }); } });

Submitted September 07, 2018 at 09:01PM by ConsistentPin

No comments:

Post a Comment