Hi allI would love your thoughts on the items below.I am building a product which analyses log information.In a nutshell a large number of log data will be sent to a load balancer, which will then be distributed to a number of servers for processing, and finally get stored in a database.Option 1The load balancer sends the log data to a few single core servers. The node app on each server processes the data (cleaning, extracting relevant data, etc.), and either inserts or updates into a MongoDB database.Option 2The load balancer sends the log data to a few single core servers. The node app on each server encrypts the data and passes it to a message queue (e.g. RabbitMQ) run by a third party (e.g. CloudAMQP.com).Another bunch of servers takes the the data off the queue, decrypts it, processes it, and inserts or updates into a MongoDB database.~~~~~~~~There are pros and cons to both.Main pro of option 1: much more simple infrastructure.Main pro of option 2: more logical setup (i.e. each node app has a clear responsibility - producer or consumer).I like option 2 the most as it seems logically correct to me, and I like the idea of the first node app just grabbing the data and passing it on, rather than having to worry about any processing.However my worry is I am overcomplicating things, and in fact maybe the first node app has roughly the same performance in both options (process, insert into database vs. encrypt, push to queue), so it may make sense to choose option 1.Any thoughts appreciated.Thanks!
Submitted May 30, 2017 at 06:36PM by just_tech_stuff
No comments:
Post a Comment