Node, ECS, and Redis. Need a worker to save huge redis data into memory?

We have a node server on Amazon ECS that gets hit around 30,000 times per second at max peak times. Since javascript is single threaded, we don't want to block the event loop. So we need to create a worker that periodically fetches 11 million rows of data from Redis and loads the data into memory. This may take a while, so we want to create a worker so there's a different event loop. How can we do this with ECS? Can we just use an npm or do we have to do anything special like setting additional stuff up on the Amazon console to have a separate worker?Furthermore, is this a good design pattern? Do I need a worker to do this? If I use a worker in Node, after I query the 11 million rows of redis data and save this object into memory, can the main server (main event loop) access this object?The previous design pattern was: for every time out server is hit, we would query one key in the Redis database. This operation takes 100 ms, which is too slow for our purposes. Hence we want to load the entire 11 million row database every 5 minutes and save it into the server's memory. So we wouldn't need to make an additional round trip to call Redis. Do we need a worker to do this redis loading operation?

Submitted April 24, 2017 at 01:30AM by thejavascripts