We are currently facing issues with the scalability/performance of Cron jobs.
We have setup 16 tracking queues in Redis, and 16 cron jobs to process the contents from them.
Threshold no. Of requests where processing starts - 10
16 Cron Jobs are configured to run for every 2 mins, in crontab.
So, for 5010 (500 iterations * 10) requests, it is taking only 16 sec, IF there is only one queue getting processed . But for 16 queues, it is taking around 4 mins (since they are run in parallel).
We have also implemented the Opcache in PHP for the Cron jobs command line (enable_cli=1), even then it is not helping with improved performance.
Is there anything else we should be doing to address this?
In case if Redis doesn’t scale, only option we see is to setup twemproxy.
Thanks in advance for your help.