Hey all!I’ve developed a node system that pulls data from multiple high traffic websocket channels, and am running into a backpressure problem.On each websocket payload receipt from various classes that talk to the websockets, I emit an event with said payload, the events then being received by a single class that then pushes the message payload onto a stream. This stream is then piped to another stream that throws the message onto a Kinesis stream via the kinesis-streams package. I did see that the author of that package switched to Kafka but thus far I’ve been happy with it and fine with the java dependency.I believe the backpressure and resulting crash from being out of memory is the stream that is piped to the Kinesis writer. The Kinesis writer is batched though, and have tried a highwatermark at various levels; same result. My questions:1) is emitting all the websocket messages as events (to be received by the single class) a bad idea? Am I possibly filling up the event bus? 1a) should I just expose the input stream to the websocket pulling classes instead of emitting as events? Can multiple classes write to a single input stream?I’ve been googling all over the place but haven’t really found another case of a successful massively high traffic data stream implementation in node.. just others with similar problems.Thank you!
Submitted June 30, 2019 at 01:17PM by favabeanzzz
No comments:
Post a Comment