I’m attempting to import Google Analytics data into Matomo for the past few years. The timestamps were progressing and now I’m receiving a timeout in the logs, eg:
This:
INFO [2020-03-19 04:23:45] 7445 Archived website id = 2, period = month, 0 segments, 990 visits in months included in: 2017-10-31,2018-02-09, Time elapsed: 3.937s
And this:
Daily rate limit reached, try again tomorrow. (Note: GA by default sets a daily limit on the number of API requests made each day to 50000. It looks like you've reached this limit. Continue the import tomorrow.)
So this seems somewhat normal, but in Matomo itself I’m not seeing how to retrieve the old data details and am seeing an error:
Is this what you’re referring to? I did notice the following in the log after the rate limit and not sure if this exception is just part of the GA limit:
[Piwik\Plugins\GoogleAnalyticsImporter\Google\DailyRateLimitReached]
Daily rate limit reached, try again tomorrow. (Note: GA by default sets a daily limit on the number of API requests made each day to 50000. It looks like you've reached this limit. Continue the import tomorrow.)
Exception trace:
() at /var/www/matomo/plugins/GoogleAnalyticsImporter/Google/GoogleAnalyticsQueryService.php:177
Piwik\Plugins\GoogleAnalyticsImporter\Google\GoogleAnalyticsQueryService->gaBatchGet() at /var/www/matomo/plugins/GoogleAnalyticsImporter/Google/GoogleAnalyticsQueryService.php:100
Piwik\Plugins\GoogleAnalyticsImporter\Google\GoogleAnalyticsQueryService->query() at /var/www/matomo/plugins/GoogleAnalyticsImporter/Importers/VisitorInterest/RecordImporter.php:59
Piwik\Plugins\GoogleAnalyticsImporter\Importers\VisitorInterest\RecordImporter->queryVisitsByDuration() at /var/www/matomo/plugins/GoogleAnalyticsImporter/Importers/VisitorInterest/RecordImporter.php:34
Piwik\Plugins\GoogleAnalyticsImporter\Importers\VisitorInterest\RecordImporter->importRecords() at /var/www/matomo/plugins/GoogleAnalyticsImporter/Importer.php:386
Piwik\Plugins\GoogleAnalyticsImporter\Importer->importDay() at /var/www/matomo/plugins/GoogleAnalyticsImporter/Importer.php:347
Piwik\Plugins\GoogleAnalyticsImporter\Importer->import() at /var/www/matomo/plugins/GoogleAnalyticsImporter/Commands/ImportReports.php:213
Piwik\Plugins\GoogleAnalyticsImporter\Commands\ImportReports->execute() at /var/www/matomo/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:257
Symfony\Component\Console\Command\Command->run() at /var/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:874
Symfony\Component\Console\Application->doRunCommand() at /var/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:195
Symfony\Component\Console\Application->doRun() at n/a:n/a
call_user_func() at /var/www/matomo/core/Console.php:140
Piwik\Console->Piwik\{closure}() at /var/www/matomo/core/Access.php:644
Piwik\Access::doAsSuperUser() at /var/www/matomo/core/Console.php:141
Piwik\Console->doRunImpl() at /var/www/matomo/core/Console.php:93
Piwik\Console->doRun() at /var/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:126
Symfony\Component\Console\Application->run() at /var/www/matomo/console:32
googleanalyticsimporter:import-reports [--property="..."] [--account="..."] [--view="..."] [--dates="..."] [--idsite="..."] [--cvar-count="..."] [--skip-archiving] [--mobile-app] [--timezone="..."] [--extra-custom-dimension="..."]
^[[A^[[B^C
root@localhost:/var/www/matomo/tmp/logs# tail -f gaimportlog.2.matomo.domainnn.tld.log
Piwik\Console->Piwik\{closure}() at /var/www/matomo/core/Access.php:644
Piwik\Access::doAsSuperUser() at /var/www/matomo/core/Console.php:141
Piwik\Console->doRunImpl() at /var/www/matomo/core/Console.php:93
Piwik\Console->doRun() at /var/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:126
Symfony\Component\Console\Application->run() at /var/www/matomo/console:32
googleanalyticsimporter:import-reports [--property="..."] [--account="..."] [--view="..."] [--dates="..."] [--idsite="..."] [--cvar-count="..."] [--skip-archiving] [--mobile-app] [--timezone="..."] [--extra-custom-dimension="..."]
Thank you so much - you were right, I had forgotten that fpm requires restarts to reinitialize php.ini vars, after doing /etc/init.d/php7.2-fpm restart with the memory increase to 512M the graph is rendering…
No more errors in the log now, so I suppose we’re just stuck waiting for the Google Analytics API limit issue. Is there a way to pull specific data for the past that has been imported thus far?
I’m guessing I’m not familiar enough with Matomo yet to drill into those old dates.
Hmm, thank you… is there a way to find out what data I’m actually getting? A way to see the most detailed info possible that’s coming in without GA 360? I’d like to see if what we are getting is “enough” or if we need to go further.
And are you saying that 360 would unlock the ability of Matomo to pull in the details, and if so, how much details?
The best way is to probably look through all reports in Matomo.
No, I have just heard that it is possible to get the raw data, but then someone would need to write a plugin that imports this data (which might be easy or complicated). See here:
I thought I was allowing GA Importer to run over the last few days here, however when I check the log at /var/www/matomo/tmp/logs/gaimportlog.2.matomo.domain.tld.log it still shows the last time the importer ran was on 2020-03-19 with the last error about reaching the API limit.
It should automatically resume each day, if not that’s certainly unexpected. There should be a button in the status table in the admin page that lets you resume manually if you are using the latest version:
