Friday 20 September 2019

Handling requests on Express server

I have a simple Express API service with a few endpoints. The APIs are used by the frontend to read/write data from Elasticsearch (ES). The said endpoints are described below.1. /api/storeCities # parses a CSV and writes 100,000 documents to the cities index in ES 2. /api/fetchCities # fetches 100,000 documents from the cities index in ES 3. /api/fetchStates # fetches 100,000 documents from the states index in ES 4. /api/fetchCountries # fetches 100,000 documents from the countries index in ESQ1. For all the fetch endpoints above, I have no problem/latency fetching data into the UI. Since node supports 1000s of concurrent requests, there shouldn't be any problem multiple users calling the same API (?).Q2a. Whenever someone calls the /api/storeCities endpoint, node parses a CSV file and writes data to ES. However during this process, all other endpoints stop responding and there's huge lag from all the fetch endpoints. Since node is non-blocking, why is the WRITE operation affecting the READ ones? Is this a problem on the ES front?Q2b. I am also using the node Cluster module so there are multiple instances of my Express app. However, the lag still persists when someone calls the /api/storeCities endpoint. It seems like node is blocking all the READ calls from ES until the WRITE call is finished.I am unable to pinpoint where the problem is, node or ES?

Submitted September 20, 2019 at 01:11PM by vanillacap

No comments:

Post a Comment