Friday 25 August 2017

Need help with cheerio and node scraping

Hi guys, so a few weeks ago I started getting into coding and made a discord bot for a game that I play.It mostly does very simple stuff like flip coins and roll dices, but now I want to add some functionality such as getting a list of daily quests from the game API.I looked up some tutorials about web scraping and tried using Cheerio. It didn't work and it might have been because it was a site that took data from a server or something, so I used phantomJS too. That didn't work out and left me even more confused.I found an API endpoint website that gives me the data for the daily quests and decided to go back to using cheerio. Of course, it still doesn't work. The API site point is http://ift.tt/2wvSkOl is the bits of codes that I have tried:Code starts herevar request = require('request');var cheerio = require('cheerio');request('http://ift.tt/2wFIgmz', function(err, resp, body) {var $ = cheerio.load(body);var dailyData = $('pre');var dailyDataText = dailyData.text();var allDailies ={dailyData: dailyDataText}console.log(allDailies.length)EndThe .length is just to see if anything was actually added into the array.which gives me an error: UnhandledPromiseRejectionWarningI also tried this:Code starts hererequest('http://ift.tt/2wFIgmz', function(err, resp, body){if(!err && resp.statusCode == 200)$('pre').each(function() //this is the element that I'm calling. Don't know if it's right andI don't have a reference either.{var url = $(this).text();urls.push(url);});EndThis actually gives me a return array length of 0, which means nothing have added.The site that I am getting the data from is very simple, and there are no id or class elements to it, so I didn't know what to call. I ended up using
 but as you know, it's not working.I'm not sure if this is not working because the site gets it's data from a server that generates the data daily or because I'm doing something wrong.Any help to solve this issue is appreciated!

Submitted August 25, 2017 at 09:57PM by HighestHand

No comments:

Post a Comment