High pagehit [50M+/month] Installation Redis Cache Bottleneck


(Florian Kaufmann) #1

Hi everybody,

we run a load balanced Matomo Installation with more than 50M Page hits per Month.

We have set up a separate Redis Server Instance for Tracking Request Queing as well as a Cache Backend as described in How to configure Redis as a shared cache in a load balanced environment? - Analytics Platform - Matomo

Just recently we noticed that the Network Link of the Redis Instance can become utilized with more than 100 MB/s outgoing data and ~ 2 MB Incoming data. The Redis Instance becomes a Bottleneck in the System.
If we move from a shared Redis Cache to a File based Cache Backend the Link saturation immediately dropps significantly

Is this normal that it needs to cache this much data and are there any suggestion to improve this setup ?
What are the Drawbacks to using file caches on every loadbalanced tracking request server against the recommendations of the faq ?
Would it help to chain in the file cache (array -> file -> redis-server) in the chained setup ?

Thank you very much in advance for any tipps.


#2

Yes that’s possible. You can chain the file caches, however, the problem is you need to synchronize the deletion of file caches across servers. When you have several servers, the file cache would be only deleted on one server but not another. I’d recommend to get in touch with www.innocraft.com where they possible have some features around that.


(Florian Kaufmann) #3

Hi Thomas,

thanks for the quick reply. Can you expand on if and how often the file caches get deleted ? Is this only done manually (e.g. Cache Buster Plugin) or is this done by matomo itself regularily ?

Also what kind of information (Dashboard and Report as far as i know but is Tracking also impacted) is put in this cache and what functionality is impacted by having a separate cache per server ?

Thanks,
Florian


#4

Matomo itself deletes those caches regularly. There are many different caches with different TTLs. Some of them have a TTL of 5 min, some 1-4 hours etc. Also they get deleted based on certain actions in the UI and if files are then not deleted across servers, there will be some random problems occurring very likely. Yes tracking is also using this cache (especially tracking, important for fast tracking) plus the reporting parts.


(Florian Kaufmann) #5

Hi Thomas,

thank you for the detailed view into the caching layer. We will experiment a bit with chaining in the file cache. The caches should be eventual consisten since their ttl runs out at different points.

I will also reach out to innocraft if possible to see if they have additional recommendations.