What about a combination of zookeeper and in-memory caching for web-farm/cluster scenarios?

When choosing caching locations, people either choose in-memory solutions, or out-of-process solutions (redis, etc).When running a web-farm/cluster setup, people almost always choose out-of-process, for obvious reasons.There are still drawbacks for using out-of-process.Network latency. You are still making network requests, but to a fast end process.Serialization. Send data has to go over the wire, you have to worry about serialization (json, protobuf, etc).Probably more, but this makes my point.The main problem with using in-memory solutions for a web-farm/cluster, is that different instances may have different/stale data. So, you have to worry about load balancing and sticky sessions. Even then, there are other issues that would have to be dealt with.What about using this caching pattern for in-memory providers in cluster scenarios:Add data to the cache using in-memory provider. Don't worry about serialization, async, etc.When a request is made to clear the cache, send a global event to all instances (using ZooKeeper, or something similar) to flush the local in-memory cache.There may be issues with this approach, in that some memory providers use local timeouts for expiring data. This would potentially cause data in-consistency issues, but this wouldn't be a problem as long is proper care is given when updating data that may be cached (clearing the cache manually).I've never seen this approach used anywhere, so I think I am missing something that others haven't.What do you guys think?

Submitted April 12, 2017 at 09:13PM by theonlylawislove