I’m facing a lot of traffic recently which has significantly slowed down my webserver, Queued Tracking has been mooted as a way forward (thanks to @rvdbreemen for the insight). I’ve been researching this and it seems simple enough to work in… Once you have a Linux environment and Redis installed.
I know very little about Linux and even less about Redis so is this may be an issue (or maybe not), it looks simple enough to install and setup so i’m thinking i just bite the bullet, spin up a linux instance on EC2 and go for it and learn on the fly. But before I do;
Has anyone used REDIS successfully with Windows using one of the unofficial msi’s, or are they to be shied away from?
Not Matomo specific, for any Linux pros’ out there who want to give some advice to a newbie; if i am spinning up a private linux EC2 instance, what should i be installing alongside Redis? is there a set load of scripts that need to be kicked off?
I know nothing about Windows, so I can’t help you there.
And you don’t have to use Redis for Queued Tracking. You can also store the temporary data in a MySQL table, but redis is of course faster (and in my opinion not hard to set up).
Redis is really simple (from a user perspective). There should be nothing to set up, just installing it and pointing Matomo at it should be enough.
Keep in mind that there is no authentication in Redis (technically there is, but it is just a hardcoded password) so everyone who can access redis can read and write everything. So you should make sure that it is only reachable from the server running Matomo. (e.g. if both are on the same server, by binding Redis only to 127.0.0.1)
So once you have this set up, you only need to specify the redis host in Matomo, the database (by default the existing 0, but then pay attention to not store any other redis data in this database) and it should work.
Many thanks for the response it was a big help I’ve got to the point where I’ve setup a Linux server, installed Redis and setup a persistent database, the sole job of this entire server is to process the matomo queues - there’s nothing else on it. It’s sat behind a firewall on a private server and limited the input to a whitelisted logon SSH and the Redis port from my webserver (thanks for the heads up on security!). I was just wondering if you have a few tips on the following;
What hard disk space should i be looking at on the disk I’ve mounted for the Redis DB? Currently i’ve got 100GB which may be way way too much as it seems super lightweight.
I’m running currently on 4 CPU and 16GB RAM so i’m going to set 4 queue workers in the plugin
If the matomo webserver, RedisDB and data server are all on separate environments, is there anything different I need to connect the RedisDB to the data server? Or will it just take the db credentials from the matomo config.ini
Cheers! would really appreciate any info or insight which might help
Steve
I am using Matomo on the ~1000 page views per month scale (and up to 10x of this sometimes), so I can’t give you that many scaling tips. (I am using the default Debian config and never changed anything)
Matomo keeps everything in Memory and saves it on disk for safety, so disk space won’t be that high.
As long as Matomo can reach redis, you just need to specify the host and port in the Plugin settings.
Cheers mate, so to confirm - does the Redis DB write directly to the mySQL database?
I just need to ensure the MySQL port is open to traffic from the Redis DB instance.
Redis has nothing to do with MySQL. It is just another database store (more precisely an key-value storage) like MySQL where Matomo can store and retrieve data.
Therefore only Matomo needs access to Redis (and MySQL)
i’m all setup as of yesterday and its working pretty well so far! Some notes;
Its dropped the CPU utilization on the webserver from 40-50% down to 8%-15% which is incredible!
This was the main reason for installing this plugin so overall i’m very happy
It has made minimal impact on the loading time on our tags though, i’m assuming this is down to the Redis server being located on a seperate server so the webserver so it is still opening a connection per tracking request.
@Lukas, is there anything i can tweak in the setup to try and reduce the tag loading time, might increasing the number of requests in one batch speed this up?
Same as we saw, the queued tracking on the webserver due to no longer processing the tracking but putting it into redis lowers the cpu usage. We do not see the seperate server as a problem, also we moved the processing to another server, and that is where the CPU now burns cycles to do the tracking processing and putting the databse int the tags.
What is the loading time of your tags btw?
Kind regard,
Robert
So our avg response times are now down in the network delay, around 11-12 ms per event. The database backend is no longer the real limitation. So the roundtrip went down by about a factor 10, before we had the avg around 150ms. So yeah, we saw a lot of improvement by using this.
Question, currently we have a single Matomo server and a separate DB and Redis server. If we moved to a load balanced Matomo server - does that have any implications on the Redis setup?
Is there anything that needs to be changed or will the current setup work fine on a single Matomo server as on a load balanced Matomo environment?
I’ve got to the point where I’ve setup a Linux server, installed Redis and setup a persistent database, the sole job of this entire server is to process the matomo queues - there’s nothing else on it. It’s sat behind a firewall on a private server and limited the input to a whitelisted logon SSH and the Redis port from my webserver (thanks for the heads up on security!). I was just wondering if you have a few tips on the following;