GeoIP2 + libmaxminddb + MaxMind-DB-Reader-php C Module Performance


(Andreas Schnederle-Wagner) #1

Hey there,
I installed latest Tar-Balls of libmaxminddb + MaxMind-DB-Reader-php C Module yesterday to get better Performance for GeoIP2 Lookups. Install worked seamless - Matomo shows me “This location provider is speeded up by the installed maxminddb extension.”
But since it’s installed - it seems like the Archive Cronjob is running longer with more CPU usage … ?
Any ideas on this?

php -v
PHP 7.2.8 (cli) (built: Jul 19 2018 11:58:32) ( NTS )
Copyright © 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright © 1998-2018 Zend Technologies
with Zend OPcache v7.2.8, Copyright © 1999-2018, by Zend Technologies

uname -a
Linux xxx.futureweb.at 3.10.0-693.21.1.vz7.46.7 #1 SMP Fri Apr 20 17:42:34 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

thx, bye from Austria
Andreas Schnederle-Wagner


(Lukas Winkler) #2

I guess only @SteveG knows enough about GeoIP2 to have an idea what could be causing this.


(Stefan Giehl) #3

That’s interesting. Actually the archiving should not do any geolocation lookups, as that should be done while tracking only.
Did you do any updates to Matomo as well?


(Andreas Schnederle-Wagner) #4

weird - nothing changed except libmaxminddb + MaxMind-DB-Reader-php C Module on this Host … maybe it’s just a weird coincidence … will check if I find other trails what caused this raise in Load Avg.


(Andreas Schnederle-Wagner) #5

seems like Traffic between Piwik <–> Piwik Database Server increased the same time.
Will check Logs if there is a Problem with Archiving somewhere … mhhh …

INFO [2018-08-17 09:39:56] 14195 ---------------------------
INFO [2018-08-17 09:39:56] 14195 SUMMARY
INFO [2018-08-17 09:39:56] 14195 Total visits for today across archived websites: 6031
INFO [2018-08-17 09:39:56] 14195 Archived today’s reports for 1107 websites
INFO [2018-08-17 09:39:56] 14195 Archived week/month/year for 1107 websites
INFO [2018-08-17 09:39:56] 14195 Skipped 0 websites
INFO [2018-08-17 09:39:56] 14195 - 0 skipped because no new visit since the last script execution
INFO [2018-08-17 09:39:56] 14195 - 0 skipped because existing daily reports are less than 3600 seconds old
INFO [2018-08-17 09:39:56] 14195 - 0 skipped because existing week/month/year periods reports are less than 3600 seconds old
INFO [2018-08-17 09:39:56] 14195 Total API requests: 4434
INFO [2018-08-17 09:39:56] 14195 done: 1107/1107 100%, 6031 vtoday, 1107 wtoday, 1107 wperiods, 4434 req, 2094677 ms, 123 errors.
INFO [2018-08-17 09:39:56] 14195 Time elapsed: 2094.677s
INFO [2018-08-17 09:39:56] 14195 ---------------------------

INFO [2018-08-17 09:39:56] 14195 Error: Got invalid response from API request: ?module=API&method=API.get&idSite=1&period=year&date=last3&format=php&trigger=archivephp. Response was ‘a:2:{s:6:“result”;s:5:“error”;s:7:“message”;s:113:“Mysqli prepare error: Table ‘piwik.piwik_archive_blob_2018_02’ doesn’t exist in engine - caused by plugin Actions”;}’

piwik.piwik_archive_blob_2018_02 is affected by: INNODB Table corruptions

Could those Errors trigger hgher Load / Traffic? mhhhh


(Andreas Schnederle-Wagner) #6

alright - it was a real unlucky coincidence … seems like the piwik.piwik_archive_blob_2018_02 table got corrupted yesterday - and because of this Archive Cronjob didn’t finish properly and started reprocessing all & everything since last successful execution …
Archive Process start: “Archiving was last executed without error 1 days 0 hours ago”
Replaced broken Table with older one out of our Backups - now Archive Cronjob won’t reprocess every Website every Hour again … :wink:

Feature Request:

  1. don’t reprocess all Websites if archiving fails fore some specific Websites (in my Case - 123 Websites out of 1107 triggered Error because of broken Table) - but all 1107 were reprocessed every Hour - maybe it’s possible to only reprocess the Websites which triggered the Error …
  2. E-Mail Notification of Admin if Archive Cronjob fails with Error (Wouldn’t have noticed that Error for a long time if - by pure coincidence - I wouldn’t have installed libmaxminddb + MaxMind-DB-Reader-php C Module the same day and closely looked into performance metrics because of that … :roll_eyes:)

Archive Cronjob - Behaviour + Error E-Mail Notifications
(Lukas Winkler) #7

At least this feature is provided by cron out of the box. Whenever a cronjob fails, it emails root. You can choose another email address by adding

MAILTO="youraddress@example.org"

in the first line of the crontab.
This of course assumes your server is able to send emails.


(Fabian Dellwing) #8

This is not fully correct. The cron.d will send the whole output via email, even if everything works fine.

Often you can use >/dev/null to get only errors, but not sure if the archiver output errors to the error line or not?

It does not need to be in the first line, you can even use multiple MAILTO in your crontab. Every line below a MAILTO will be send to this MAILTO until EOF or another MAILTO is reached.


(Andreas Schnederle-Wagner) #9

it’s already configured like this … mhhh …

MAILTO="mymail@futureweb.at"
5 0-4,6-23 * * * /usr/bin/php /var/www/piwik/console core:archive --url=https://domain/ > /var/log/piwik.cron.log

Guess the redirection to Logfile prevents sending the Mail … any idea how to achieve booth? Log + Error Mail?


(Fabian Dellwing) #10

Do a grep on the log?

0 0 * * * grep 'Error' /var/log/piwik.cron.log

This should send you an email every day at 0:00 if the latest archiving contained any error.


(Lukas Winkler) #11

I am using a redirect and already got notification mails by cron, so it seems like at least some errors end up on stderr.

ERROR [2018-07-16 00:00:32] 30367  [SearchEngineKeywordsPerformance] Error while importing Bing crawl stats for https://lw1.at/: ERROR!!! ThrottleUser

(Fabian Dellwing) #12

I see the difference there:

Your line is prefixed with ERROR and the invalid response error is INFO. I think this should not be that way…?


(Fabian Dellwing) #13