Saturday, 14 July 2018

How to filter large amounts of data (react/node)

Hey,before I will talk about the challenge I'm facing, I will describe my project as simple as possible:Backend (Rest API): node.js (express.js), postgreSQLFrontend: React.jsThe frontend can retrieve data from the backend via HTTP queries. One possibility is to retrieve binary data (blob). In this case only the path is stored in the db (which references the file), the file is stored on a filesystem.The challenge I'm facing is the size of the data, where a file can be up to 5MB and the file content consists of ~15.000 lines. Via ReactVirtualized InfiniteLoader displaying the data becomes super fast. Now I would like to be able to filter the file content on the front end: the user can type in text/regexp into a form(input) and on this basis, the file content will be filtered, meaning the the file content will be displayed, but filter matches will be omitted, e.g:File (pre)aaa bbb ccc Filter:aaa File (post)bbb ccc However I`m unsure how to accomplish this? Setting up the filter is easy, but should I do this on the frontend (client) or create a route on my backend, which would take the filter as a query param, filter the data on the server and send it back to the frontend.Any thoughts, tips and recommendations?Thank you!

Submitted July 14, 2018 at 05:32PM by Fasyx

No comments:

Post a Comment