Wednesday 29 January 2020

Efficient way to scan just the first few lines from large files?

I need to scan through a lot of text files that are around 1MB or more. I just need to parse one short string value from a line that should appear in the first few lines of the file. I thought it would be best to read line by line until I find the value rather than loading the entire file. Is this the fastest way to do that that will safely close the file & line reader?function parseHeader (filename) { return new Promise(resolve => { let value // will hold value when found const input = fs.createReadStream(filename, 'utf8') const reader = readline.createInterface(input) reader.on('line', line => { if (!line.startsWith('value=')) { return // Not the line we want, continue. } value = parseValue(line) // Got the value, so cleanup reader.close() input.destroy() resolve(value) }) reader.on('close', () => { if (value == null) { // File was closed before finding line resolve(undefined) } }) }) } This only needs to run on Linux, so I shouldn't need to deal with any cross-platform filesystem issues.

Submitted January 29, 2020 at 06:22PM by spacejack2114

No comments:

Post a Comment