Tuesday 31 October 2017

Resolving a promise and writing some data to CSV, why are my rows writing to the file before my header?

Hey folks, I am currently trying to write a reddit scrapper so that I can learn Node. I am writing some backend code and I am used to synchronous code execution (I come from ruby/rails) so Node is throwing me off a bit.I am using snoowrap as my reddit API wrapper. I first query for my dailyThread and I attempt to resolve the returned promise so that I can grab some data and spit it out into a csv. The weird thing is that I append my header first and then the rows of data, but sometimes, my row of data appears before my header.i.e.fs.appendFile(filePath, header, (err)) gets called before fs.appendFile(filePath, row, (err), but row sometimes appear before header.What am I doing incorrectly? (Should I be making the entire thing synchronous and if so, would this defeat the whole purpose of using node/js?)In production apps, how do most companies/teams handle tasks that need to be synchronous? Are they just calling a service and having that service execute synchronously and letting Node just be a controller? Promise.resolve(dailyThread).then(result => { let threadId = result[0].id console.log(threadId) reddit.getSubmission(threadId).expandReplies({limit: Infinity, depth: 1}).then(thread => { let threadName = 'Daily General Discussion - ' + date let subreddit = 'ethtrader' let header = "threadDate, score, body, sourceId, subreddit, threadName \r\n" let filePath = `csv/reddit_${dateFile}.csv` try { if(fs.openSync(filePath,'r')) return } catch (err) { fs.appendFile(filePath, header, (err) => { if (err) throw err console.log('Write Header:' + threadId) }) let mapping; mapping = thread.comments.map(c =>{ let body = c.body.replace(/(?:\r\n|\r|\n)/g, ''); let score = c.score let sourceId = c.id let row = date + ", " + score + ", " + body + ", " + sourceId + ", " + subreddit + ", " + threadName + "\r\n" fs.appendFile(filePath, row, (err) => { if (err) throw err console.log('Write Row:' + sourceId) }) }) } }) })

Submitted October 31, 2017 at 04:15PM by Guy-Lambo

No comments:

Post a Comment