Core:archive script is taking too long and php is consuming about 10GB memory

Hello!!

I have a website ID where logs of 140 vhosts go in real time. Server is with 12 cores and 16GB od RAM.
When it comes time to archive the report, the process takes quite a while and php uses almost 10GB of memory. Finally I get an error:

WARNING [2020-03-25 11:11:31] 32051 /usr/share/matomo/core/CliMulti/Process.php(264): Warning - shell_exec(): Unable to execute ‘ps ex 2>/dev/null | awk ‘! /defunct/ {print $1}’ 2>/dev/null’ - Matomo 3.13.3 - Please report this message in the Matomo forums: https://forum.matomo.org (please do a search first as it might have been reported already)
ERROR [2020-03-25 11:11:31] 32051 Got invalid response from API request: ?module=API&method=API.get&idSite=70&period=month&date=last1&format=php&trigger=archivephp. The response was empty. This usually means a server error. A solution to this error is generally to increase the value of ‘memory_limit’ in your php.ini file. For more information and the error message please check in your PHP CLI error log file. As this core:archive command triggers PHP processes over the CLI, you can find where PHP CLI logs are stored by running this command: php -i | grep error_log

and the process brakes on period = month.

What can i do for optimize this process to take longer, and use less memory. Can any of this parameter help me:

-force-all-websites
–force-all-periods .
–force-timeout-for-periods
–skip-idsites
–skip-all-segments
–force-idsites
–force-periods
–force-date-last-n
–force-date-range
–force-idsegments
–concurrent-requests-per-website
–disable-scheduled-tasks
–accept-invalid-ssl-certificate
–xhprof
–help (-h)
–concurrent-requests-per-website
–concurrent-archivers

So far i have skipped the website id in core: archive scripts.

Any ideas? Thank you for your help!

Hi Billy,

To me it looks like your php process is running out of memory, normally default is 256MB, that is not enough in my experience. We decided to to 8GB on php memory limit.

We are running a high volume site. Just a few questions for you:

  • what’s the volume of events per day?
  • are you running a single server with a database?
  • have you setup the php memory limit to 8GB yet?

The core:archiver process is very compute and memory intensive, so I resorted to 8GB memory limit on our archiver server. We separated the tracking server and the archiver server and the database ser
You should read this: How to configure Matomo for speed User Guide - Analytics Platform - Matomo

You are asking if any of the parameters can help. The simple answer, start setting up your server and all php and database as good as you can, before trying to tune the matomo app itself. As the proper functioning of the application is really only possible once you have setup everything properply.

Following the performance tuning (link above) does help a lot,

Kind Regards,
Robert

Hi

i was rising memory limit in past few days from 4GB to 8GB but arhiving process newer ends :slight_smile:

1.Visits Summary (biggest website, ID 70)
82,860 visits, 82,676 unique visitors
7 min 14s average visit duration
44% visits have bounced (left the website after one page)
18.4 actions (page views, downloads, outlinks and internal site searches) per visit
0 average generation time
529,733 pageviews, 464,358 unique pageviews
386 total searches on your website, 221 unique keywords
993,291 downloads, 931,913 unique downloads
0 outlinks, 0 unique outlinks
18,316 max actions in one visit

  1. yes, it is single server with database on other partition
  2. yes, i ican put more but i would rather optimize something

Ok, i have 12core CPU and 16GB RAM Debian 9, apache 2.4, php7, mariadb10.1. I have over 50 sites who are schedule to import logs with import_log.py (just 2 parameters --recorders=4 --recorder-max-payload-size=1000, without bots, redirects, etc…) over night in diferent time.
The largest site is that one with ID 70 which receives logs of 140 vhost in real time (CustomLog directive) and that works fine till it come archiving.
Im not using NFS, i disabled web archiving, set innodb_buffer_pool_size to 50% of RAM for now, tuned up apache workers. Dont know if php caching (opcache or fpm?) might help in my case for archiving proces.

Sorry to jump in on this but i think this may be affecting me as well

when you guys say “have you setup the php memory limit to 8GB yet” is that the memory_limit in php.ini?

Apologies i’m a total PHP noob

Cheers

yes memory that php is using

1 Like

@Stevie: Yes, it’s the php.ini memory_limit parameter that you need to set. Make sure it is set on the php.ini that is used in the apache and the commandline (sometimes you have different versions on a single machine.

@Billy:
So it’s not a very big number of events and still you stall the php archiver process. Have you checked the tmpfs space of the database? If the database is queried, then the tmpfs is used. If that space runs out, query’s hang forever, until the tmp space clears out.
I had my database with way to small space. Do not use a memory based tmpfs, as that will run out of space easily.

@rvdbreemen
you think on tmp_dir, max_tmp_tables, tmp_table_size? I have that options i dont use RAM based tmpfs but matomo recomends it:
" We recommend to configure MySQL and point the tmpdir to a filesystem in the RAM for example using tmpfs, instead using a tmp directory on the disk. As MySQL uses tmpdir to create and store temporary datasets, it brings additional speed when only the RAM needs to be used."

I didn’t mention the options I had to include :):
https://matomo.org/faq/how-to/faq_54/
This is the reason for the extremely poor performance. It is very important to me if there is an option to create an archive with these options for only one website ID and not for all. these options are global and set in config.ini

Do you have experience with following options:

–concurrent-requests-per-website
–concurrent-archivers

@glatzenarsch
I know that it say’s you can use it. But the problem is that once you go that route you need to be very aware how the tmpfs is used. We used the tmpfs initially too, as described in the manual.

The tmpfs can be used to speed up things in RAM, that is for sure. However when running concurrent archivers and concurrent request per website, you can run out of tmpfs space. So either you monitor that very closely or you run less concurrent archivers.

tmpfs is used to store intermediate resultats from queries, for example parts of queries that need joining.However if you have let’s say 4 concurrent archivers running and also 5 website requests per archiver, then you can have 20 concurrent queries running. All using the same tmpfs to “speed up” your work.

In case the tmpfs gets low, your queries will start to wait for each other. You can even end up deadlocking your queries completly, and the archivers just stop and wait forever.

Yes, I have seen all situation occure. So I went back to tmpfs on highspeed SSD, rather than 64GB of memory. However if you have plenty of memory, then please use RAM. But remember to monitor your tmpfs usage in that case. It should never ever be 80% full at any one point.

Also if you have high volume traffic the queries just take up a lot of tmpfs, in that case you really do not want a memory based tmpfs either. Even a single archiver with a single query could potentionally fill up your tmpfs.

And once you do, the only way to recover fast, is to reboot your database.

So my advice, do not use tmpfs, unless you can have a lot of memory to use and closely monitor your tmpfs usage.

The options you ask about:
“It is very important to me if there is an option to create an archive with these options for only one website ID and not for all.”

Yes, it possible to setup crontabs to just archive a single siteid at a time. This command does the trick:

–force-idsites

Read this too: https://matomo.org/docs/setup-auto-archiving/

This will enable you to have full control on which siteid’s are run. I recommend setting up a shell script that executes them in order. Do not try to schedule them using cron, you will assume that after a period of time the previous archiver has done its work, but you cannot assume that ever. Sometimes queries just take a very long time, it totally depends on the number of segements and volume of data that needs processing.

Do schedule the archiver work on a regular basis. I use a cronjob right now on our archiverserver that executes the processing of ALL siteids, it’s launched every 15 minutes. The console command only adds more archivers, if it is not already running the maximum number.

The options we use right now:

  • concurrent-archivers = 4
  • concurrent-request-per-website = 5

It is important to realize that this means you can have up to 4 * 5 = 20 querý’s to your database at the same time. This means your database tuning needs to be done well. This is were master - slave setup comes into play. You do not want your ingest traffic and archiver query’s running on the same database, that’s asking for problems with performance.

So if you are doing high volume, you do want to split up the database in incoming write traffic, on the master database and query’s on a read slave databse. Then on the read database you run as many io_read_threads as you can. Match them up with the number of CPU’s.

So let’s say you have a slave, running on 32CPU’s, and you run 20 queries at a time, remember, that every query needs a single CPU to perform optimal. Then you might wanna setup 20 io_read_threads and 12 io_write_threads… now lets remember, the mariadb / mysql thread scheduler needs some of its own threads. But you want the io_wait to be as low as possible on you datbase.

Then on your limits from the FAQ, yes, if you move those limits high, to the suggested 5000, that will cause a lot of processing. Also remember, when you do so, you will have that for your segements as well. You might wanna experiment a little with lower limits.

I have no clue if you can run these limits for a single siteid… maybe @matthieu could comment on that himself.

Kind regards,
Robert

Yes i agree with using tmps from disk i use it as i have no memory to throw.
And yes I’m using force site ID for the bigest site ID and for everyone else in particular, everything is from cron.
I deleted all segments, anyway if i need special searches i will create segment again, but didnt help for biggest site ID.
We have wazuh which controls matomo logs and if error ocures sends mail :).

As for the limit of 5000 we only need it for one site ID but the options are global so please @matthieu again if there is a way to apply these options on only one site or exlude on some?!
You run an archive 4 by 4 sites every 15 minutes if I understand correctly?

You have a nice setup of clusters, I’ll think about it when needed, thank you very much!