I am building an application that aggregates visualizes data about a user's social media and streaming platform accounts.A user will create an account on my app, and then connect other platforms. Once they have authorized the other platforms via OAuth, the tokens are stored in my MongoDB database.Now my issue is that I want to use those tokens to populate the database with all the pertinent data about the user's account on that connected platform. Let's take for example, a user creates an account authorizes my app to see their Mixer (streaming platform) analytics data.I then want to have a separate 'worker' application that populates my database with the Mixer analytics for that user. My front-end of my app can then query my database for all the information.The reason I want this separate app is to avoid rate-limiting. I realized pretty fast that if I want to have the analytics I planned to have, it would require hundreds of requests to certain REST APIs to create a full report on a user's Mixer account. Mixer has a rate limit of 100 requests per 3 seconds. The data is historical and will never change, so there is no reason to grab it from Mixer every time and make the user wait up to a minute each time. Honestly, this would never scale as the rate limit is tied to IP and if 100 people were using the platform at the same time, these requests would take way too long.My question, what is the best way to host this separate database worker? I use heroku for the back end of my application, and I have looked into 'worker' dynos, but I am not sure that is what I want (also, they cost about 7$ per month). I use netlify for the front end of my application. Maybe I should use Netlify functions? I, again, am not sure if that is relevant. I am not even sure what to google search to find answers, so all replies would be appreciated here.Are there any examples of something like this on github?
Submitted August 01, 2019 at 11:52PM by devaent1316
No comments:
Post a Comment