GA Importer: Error on day YYYY-MM-DD, Failed to reach GA after 30 attempts

Support & Bugs
1

New to Matomo, but really enjoying it. I’m trying to pull the historical analytics from my GA account into Matomo (hoping to “cut the cord” a little). I’ve set up the OAuth credentials properly and have been able to create two importing tasks, however:

  1. I’m not seeing any imported data.
  2. I’m getting some errors which I’m not sure how to troubleshoot further.

From one of my tasks:
errored

Error message: Error on day 2016-08-23, Failed to reach GA after 30 attempts. Restart the import later.
These errors are unexpected and will likely continue every time you run the import on this day. To resolve this issue, please ask on the forums. If you can provide access to your GA account to a member of Matomo’s support team it will provide a quicker resolution.

I noted that the link to the “forums” is not correct (has an “s” – this website is forum.matomo.org), but if someone could point me in a direction for finding out more about it, I will be very appreciative. This isn’t critical, but I’m curious about getting it to work in order to keep some of the histories.

OTHER INFO: I’ve increased the max_execution_time and the max_input_time in PHP to accommodate longer running scripts. I’ve also enabled mysqli.allow_local_infile (On) in the php.ini file.

Lastly, the console of my site reports no warnings or errors – the troubleshooting guides for setting up the site were excellent and I was able to resolve all the problems.

FINAL THOUGHT: Is there a specific port in the firewall of my server that needs to be opened for this to work properly? Right now I’ve got 443 and 80 open. Everything else is closed.

2

Hi,

You are right, I fixed this in https://github.com/matomo-org/plugin-GoogleAnalyticsImporter/pull/102

Can you maybe check your tmp/logs/ directory in Matomo? The importer should write a log file with more information there. It should contain the exact error that occurs when making a HTTP request to Google Analytics.

3

Dear Lukas,

Thank you for the response. Here is the last log entry for the import from the tmp/logs/ directory per your request. Let me know if there are other things I can provide or that I need to check.

I noticed the “CustomDimensions” plugin warning – should I add that in? I only used “basic” Google Analytics since around 2016, so I didn’t do much with it except track visits, etc.

Thanks again for your support.

-Thom

----- Log Entry Below -----

googleanalyticsimporter:import-reports [–property="…"] [–account="…"] [–view="…"] [–dates="…"] [–idsite="…"] [–cvar-count="…"] [–skip-archiving] [–mobile-app] [–timezone="…"] [–extra-custom-dimension="…"]

Import for site with ID = 3 has errored, will attempt to resume.
No dates specified with --dates, importing data from when the GA site was created to today: 2016-07-20
WARNING [2020-08-09 23:05:08] 4536 The CustomDimensions plugin is not activated or present, skipping custom dimension import.
INFO [2020-08-09 23:05:08] 4536 Using existing custom variable slots.
Importing the following date ranges in order: 2016-07-20,2020-08-08
Importing reports for date range 2016-08-23 - 2020-08-08 from GA view 125953805.
WARNING [2020-08-09 23:05:08] 4536 No GA goal ID found mapped for ‘Time on Task’ [idgoal = 1]
INFO [2020-08-09 23:05:08] 4536 Importing data for GA View 125953805 for date 2016-08-23…
Running archiving for newly imported data…
INFO [2020-08-09 23:30:18] 4536 Last archived date (2016-08-23) is earlier than last import date (2016-08-22, no need to archive for site ID = 3
ERROR [2020-08-09 23:30:18] 4536 Uncaught exception: /var/www/html/plugins/GoogleAnalyticsImporter/Google/GoogleAnalyticsQueryService.php(196): Failed to reach GA after 30 attempts. Restart the import later.
Failed to reach GA after 30 attempts. Restart the import later.
#0 /var/www/html/plugins/GoogleAnalyticsImporter/Google/GoogleAnalyticsQueryService.php(100): Piwik\Plugins\GoogleAnalyticsImporter\Google\GoogleAnalyticsQueryService->gaBatchGet(Object(Piwik\Date), Array, Array, ‘ga:sessions’)
#1 /var/www/html/plugins/GoogleAnalyticsImporter/Importers/VisitsSummary/RecordImporter.php(47): Piwik\Plugins\GoogleAnalyticsImporter\Google\GoogleAnalyticsQueryService->query(Object(Piwik\Date), Array, Array, Array)
#2 /var/www/html/plugins/GoogleAnalyticsImporter/Importer.php(432): Piwik\Plugins\GoogleAnalyticsImporter\Importers\VisitsSummary\RecordImporter->importRecords(Object(Piwik\Date))
#3 /var/www/html/plugins/GoogleAnalyticsImporter/Importer.php(369): Piwik\Plugins\GoogleAnalyticsImporter\Importer->importDay(Object(Piwik\Site), Object(Piwik\Date), Array, ‘’)
#4 /var/www/html/plugins/GoogleAnalyticsImporter/Commands/ImportReports.php(243): Piwik\Plugins\GoogleAnalyticsImporter\Importer->import(‘3’, ‘125953805’, Object(Piwik\Date), Object(Piwik\Date), Object(Piwik\Plugins\GoogleAnalyticsImporter\ImportLock))
#5 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\GoogleAnalyticsImporter\Commands\ImportReports->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\GoogleAnalyticsImporter\Commands\ImportReports), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /var/www/html/core/Console.php(140): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/html/core/Access.php(644): Piwik\Console->Piwik{closure}()
#11 /var/www/html/core/Console.php(141): Piwik\Access::doAsSuperUser(Object(Closure))
#12 /var/www/html/core/Console.php(93): Piwik\Console->doRunImpl(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/html/console(32): Symfony\Component\Console\Application->run()
#15 {main}

[Exception]
Failed to reach GA after 30 attempts. Restart the import later.

4

Hi,

No, that just means that no custom dimensions will be imported.

I don’t have a suggestion, but @diosmosis probably knows more.

5

Ok, @Lukas – thank you! I’ll wait and see if @diosmosis will join the discussion.

6

@tallthom can you enable verbose logging and start an import:

image

then check the log file that’s created (and post it)?

Can you also make sure you have both of the following APIs enabled in the google app developer console:

image
image1886×486 38 KB

? Note: if you have to make a change here, you may have to re-upload the oauth client configuration.

7

Thanks for the reply, @diosmosis! I’ll be happy to start a new import with verbose logging. However, will I be able to merge this data with my current site data? Or will this just be a separate “defunct” site when I’m finished?

I’ve enabled both of those APIs at first, but I’ll reconfirm they are enabled.

-Thom

8

@tallthom it will be a defunct site. You can limit the import to one or a couple days, we just need to reproduce the error with more logging.

9

This is interesting. Once I added the new site (same settings, just has logging turned on) – it now is importing the data into BOTH places. No errors being reported on the webpage. @diosmosis

However, this error is being reported in the log file:

DEBUG [2020-08-12 08:09:12] 20836 Waiting 60s before trying again…
DEBUG [2020-08-12 08:10:12] 20836 Google Analytics returned an error: {
“error”: {
“code”: 429,
“message”: “Quota Error: Number of recent failed reporting API requests is too high, please implement exponential back off.\nSee developer documentation for details at https://developers.google.com/analytics/devguides/reporting/core/v4/limits-quotas#reporting_apis_request_errors”,
“errors”: [
{
“message”: “Quota Error: Number of recent failed reporting API requests is too high, please implement exponential back off.\nSee developer documentation for details at https://developers.google.com/analytics/devguides/reporting/core/v4/limits-quotas#reporting_apis_request_errors”,
“domain”: “global”,
“reason”: “rateLimitExceeded”
}
],
“status”: “RESOURCE_EXHAUSTED”
}
}

10

@tallthom thanks for getting the debug logs, this is interesting: “Number of recent failed reporting API requests is too high” it looks like a new global quota limit. GA must have limited your requests in a way the plugin doesn’t detect. Quota limits are temporary, which is why they started working again. The failure might happen again, but it will always be a temporary issue.

Can you email the full debug log to benaka@innocraft.com?

11

I’ve sent the logs as requested. Let me know if there is anything else I might provide to help.

12

@tallthom From looking at the logs it seems like there were two issues going on, the first was the GA API backend repeatedly failing on it’s side, then the importer hitting a new limit based on backend errors. There’s not much the importer can do since this is due to an error in Google Analytics, but I’ve released a new version that will hopefully avoid it and better communicate if it happens again.

13

Thanks for your follow-up, @diosmosis – I really appreciate it. I’ll look for the update and report anything else new I find.