Wednesday, 6 January 2016

Problem with slow and recursive RethinkDB queries (thinky.io)

I'm trying to build a benchmark app for mongodb and rethinkdb to compare speeds between writes, reads, and join lookups for each but seem to have 2 issues with the rethink side of things:Rethink writes seem slow. I'm using the Thinky ORM because its really similar to the Mongoose format and can almost drop in directly with minor other changes to most of my apps. I can only manage about 1,000 writes per second no matter how many connections I allow or what write_ack and durability settings I enable. I feel like this is at least 10x short of what it should be able to do, if not 50x.During the Rethink read test, I attempt to lookup each record from the write test one by one using the num key from my loop. This is not an index, just a regular field, and only 1 record exists with each number. However, during this test I can see the Rethink server getting reads of up to 100,000 reads per second for several minutes and never seems to end (or I haven't waited long enough, EDIT: just noticed it finished 121 seconds later on 1/10th of the actual test data set. Also noticed it responded with a console log on a particular record 3 times when it should have only happened once.). Its almost like its checking each record in the DB to see if it matches, rather than using fancy algorithms to find the record more efficiently.A few things I've looked at:I'm aware there is a bulk insert method that is more efficient, however I am attempting to simulate high traffic on an API dealing with single records at a time.The for loops I'm using in the tests seem to be locking up the event loop rather than any kind of DB latency issues. I have also attempted using setImmediate, process.nextTick, and setTimeout to put a short delay in, however this barely helps if at all and also introduces false lag into tests which I want to avoid. I know node and the DBs are capable of some insane throughput so I cant imagine I'm at the limit at 1k writes per second.Uploaded the code to github here, would love for someone to take a look and let me know where I'm going wrong. There are some variables at the top of the index file for connection settings to DBs. Its currently set to use a docker instance for the DBs. If you test it make sure the ports are configured correctly for your instances.Gist of the program:Load index.jsLoad tests from benchmark folder for mongo and rethink in alphabetical order (forcing order with #_ prefix)Begin looping through mongo testsSetup the test (connect to DB, load models)Run the test (write, read, and joins)Save results and cleanup to prepare for next testRepeat for rethink testsOutput results, matching up results for similar tests if availableAppreciate any help!

Submitted January 06, 2016 at 08:56AM by Sythic_

No comments:

Post a Comment