I have build my own node script which should read many json files contraining arrays from one directory, and merge them to one single file. There are 1000 files at ~2500kb each, containing an array of ~350 000 numbers (numbers up to 5 000 000 000) each. I read the values, put them into a string, and after I have read 10000 values, I append the string to the file, reset the string, and continuing reading data and writing it into the string. I think that the process should take almost the same amount of RAM all the time because the files are around the same size, and after I have read the file I delete the variable holding the data from the file. And I always write the same amount of numbers to the string before appending it into the real file. And after that I reset the string. But the program is taking more and more ram for each second, until it reaches 1,5gb, and then it crashes because of that it has no avalible RAM left. I do not understand why the RAM usage would increase. Please help me. Edit: I have tried to reduce the stack size from 10000 to 5000 but it didn't had any effect.
Submitted June 22, 2017 at 04:08PM by CMDJojo
No comments:
Post a Comment