Is this a bad design?

I am working on a design for a new project. I am usually a front end dev, so I’m feeling a little out of my league here. I’ve had some help at work with the design but it still doesn’t feel right to me.Essentially:The Node server runs in a Docker container. It receives some data from a client (using Express) and appends it to a CSV file. Eventually, after some time or the file has reached a certain size, the CSV file is uploaded to S3.To scale this, we will run multiple containers and load balance them. They will all use a shared volume where they write the CSV data.The one thing I can’t find a good solution for is what to do if one of the Node processes crashes or is killed. There will be a bunch of CSV data that now will never be uploaded to S3. I suppose since they use a shared volume one of the other servers could “adopt” the CSV data and upload it, but how would I coordinate this between multiple servers?Sorry if this is a horrible design or stupid questions, like I said I am a bit out of my league here.

Submitted October 13, 2018 at 05:47PM by The_Talisman