Archive log error year fails


#1

We’re receiving these errors in the archive log even though archiving has run/complete.

"INFO CoreConsole[2014-07-07 23:25:43] [35b76] Done archiving!
INFO CoreConsole[2014-07-07 23:25:43] [35b76] ---------------------------
INFO CoreConsole[2014-07-07 23:25:43] [35b76] SUMMARY
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Total visits for today across archived websites: 809
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Archived today’s reports for 17 websites
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Archived week/month/year for 17 websites
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Skipped 0 websites day archiving: existing daily reports are less than 3600 seconds old
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Total API requests: 68
INFO CoreConsole[2014-07-07 23:25:43] [35b76] done: 17/17 100%, 809 vtoday, 17 wtoday, 17 wperiods, 68 req, 52760351 ms, 1 errors.
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Time elapsed: 52760.351s
INFO CoreConsole[2014-07-07 23:25:43] [35b76] ---------------------------
INFO CoreConsole[2014-07-07 23:25:43] [35b76] SCHEDULED TASKS
INFO CoreConsole[2014-07-07 23:25:43] [35b76] Starting Scheduled tasks…
ERROR CoreConsole[2014-07-07 23:34:11] [35b76] Got invalid response from API request: http://mysite/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&co$

INFO CoreConsole[2014-07-07 23:34:11] [35b76] done
INFO CoreConsole[2014-07-07 23:34:11] [35b76] ---------------------------
INFO CoreConsole[2014-07-07 23:34:11] [35b76] ---------------------------
INFO CoreConsole[2014-07-07 23:34:11] [35b76] SUMMARY OF ERRORS
INFO CoreConsole[2014-07-07 23:34:11] [35b76] Error: Got invalid response from API request: http://mysite/index.php?module=API&method=API.get&idSite=1&period=year&date=last7$
INFO CoreConsole[2014-07-07 23:34:11] [35b76] Error: Got invalid response from API request: http://mysite/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=$
ERROR CoreConsole[2014-07-07 23:34:11] [35b76] 2 total errors during this script execution, please investigate and try and fix these errors."

As you can see the core:archive is taking all day to run once! The longest time seems to be taken by our most visited site ‘siteid1’ - the ‘month’ takes really long & ‘year’ takes forever to complete with an error saying ‘0’ visits which is not possible.

“INFO CoreConsole[2014-07-07 08:46:23] [35b76] Starting Piwik reports archiving…
INFO CoreConsole[2014-07-07 08:46:55] [35b76] Archived website id = 1, period = day, 1069 visits in last last4 days, 170 visits today, Time elapsed: 31.952s
INFO CoreConsole[2014-07-07 09:03:48] [35b76] Archived website id = 1, period = week, 23234 visits in last last8 weeks, 175 visits this week, Time elapsed: 1013.015s
INFO CoreConsole[2014-07-07 11:16:44] [35b76] Archived website id = 1, period = month, 114920 visits in last last8 months, 2478 visits this month, Time elapsed: 7976.503s
ERROR CoreConsole[2014-07-07 16:06:47] [35b76] Got invalid response from API request: http:/mysite/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&forma$
INFO CoreConsole[2014-07-07 16:06:47] [35b76] Archived website id = 1, period = year, 0 visits in last last7 years, 0 visits this year, Time elapsed: 17402.930s
INFO CoreConsole[2014-07-07 16:06:47] [35b76] Archived website id = 1, 4 API requests, Time elapsed: 26424.402s [1/17 done]”

If I try to run a ‘year’ report from the UI (http://mysite/index.php?module=CoreHome&action=index&idSite=1&period=range&date=2013-07-01,2014-06-30#/module=Dashboard&action=embeddedIndex&idSite=1&period=range&date=2013-07-01,2014-06-30&idDashboard=1) it hangs/continues running for ages - it hasn’t given an error as yet, just shows as ‘loading’ widgets.

we are running version 2.4.0, we have cleared out log files older than 180 days old.

Thanks
Lynn


(mike) #2

Hi lynn,

I’m having this problem too and put up a post about this too: 301 Moved Permanently

I’ll be watching here to see if you get a useful reply, but on the chance I get a reply, I thought you might appreciate the link.

Mike


#3

Thanks Mike.

It looks like it’s cleared up now, but the now all my month & year data for 2013 is missing.

I deleted LOGS older than 6 months & then ran core:archive which took all day, finished last night WITHOUT ERRORS & then immediately ran a cronjob core:archive & that completed within an hour without errors.

"NFO CoreConsole[2014-07-08 07:37:26] [c8def] - Will process 17 websites with new visits since 7 days 9 hours , IDs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 1$
INFO CoreConsole[2014-07-08 07:37:26] [c8def] ---------------------------
INFO CoreConsole[2014-07-08 07:37:26] [c8def] START
INFO CoreConsole[2014-07-08 07:37:26] [c8def] Starting Piwik reports archiving…
INFO CoreConsole[2014-07-08 07:37:43] [c8def] Archived website id = 1, period = day, 661 visits in last last2 days, 112 visits today, Time elapse$
INFO CoreConsole[2014-07-08 08:05:04] [c8def] Archived website id = 1, period = week, 26643 visits in last last9 weeks, 662 visits this week, Tim$
INFO CoreConsole[2014-07-08 10:45:37] [c8def] Archived website id = 1, period = month, 98252 visits in last last9 months, 2981 visits this month,$
INFO CoreConsole[2014-07-08 13:38:37] [c8def] Archived website id = 1, period = year, 98372 visits in last last7 years, 98372 visits this year, T$
INFO CoreConsole[2014-07-08 13:38:38] [c8def] Archived website id = 1, 4 API requests, Time elapsed: 21671.908s [1/17 done]

INFO CoreConsole[2014-07-08 17:35:45] [c8def] Archived website id = 21, 4 API requests, Time elapsed: 65.387s [17/17 done]
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Done archiving!
INFO CoreConsole[2014-07-08 17:35:45] [c8def] ---------------------------
INFO CoreConsole[2014-07-08 17:35:45] [c8def] SUMMARY
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Total visits for today across archived websites: 765
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Archived today’s reports for 17 websites
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Archived week/month/year for 17 websites
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Skipped 0 websites day archiving: existing daily reports are less than 3600 seconds old
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Total API requests: 68
INFO CoreConsole[2014-07-08 17:35:45] [c8def] done: 17/17 100%, 765 vtoday, 17 wtoday, 17 wperiods, 68 req, 35899350 ms, no error
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Time elapsed: 35899.351s
INFO CoreConsole[2014-07-08 17:35:45] [c8def] ---------------------------
INFO CoreConsole[2014-07-08 17:35:45] [c8def] SCHEDULED TASKS
INFO CoreConsole[2014-07-08 17:35:45] [c8def] Starting Scheduled tasks…
INFO CoreConsole[2014-07-08 17:37:54] [c8def] task,output
Piwik\Plugins\CoreAdminHome\Tasks.purgeOutdatedArchives,Time elapsed: 105.443s
Piwik\Plugins\ExamplePlugin\Tasks.myTask,Time elapsed: 0.000s
Piwik\Plugins\PrivacyManager\Tasks.deleteReportData,Time elapsed: 0.138s
Piwik\Plugins\PrivacyManager\Tasks.deleteLogData,Time elapsed: 0.039s
Piwik\Plugins\CorePluginsAdmin\Tasks.clearAllCacheEntries,Time elapsed: 0.078s
Piwik\Plugins\CorePluginsAdmin\Tasks.sendNotificationIfUpdatesAvailable,Time elapsed: 0.013s
Piwik\Plugins\CoreAdminHome\Tasks.optimizeArchiveTable,Time elapsed: 16.231s
Piwik\Plugins\UserCountry\GeoIPAutoUpdater.update,Time elapsed: 4.148s
Piwik\Plugins\CoreUpdater\Tasks.sendNotificationIfUpdateAvailable,Time elapsed: 0.026s
INFO CoreConsole[2014-07-08 17:37:54] [c8def] done
INFO CoreConsole[2014-07-08 17:37:54] [c8def] ------


(mike) #4

If you deleted the logs older than six months, there is no source data for the archive cron jobs to process, and you’ll have nothing in your reports. Unless this was on purpose, you may have deleted some important data.

So, if that’s the case, it took all day to tally six month’s data with about 100k hits, and then another hour to run it immediately again?


#5

This is 2013 data we are talking about & we had/have a cronjob running the archiving every hour since installation - the data is there for ‘weeks’ but not ranges or days or years or months…


#6

http://forum.piwik.org/read.php?2,117570


#7

What is odd is that this appeared in the log files in the archive log after ‘deleting old log files’…'Piwik\Plugins\PrivacyManager\Tasks.deleteReportData,Time elapsed: 0.138s ’ even though ‘report data’ was NEVER set to be deleted ONLY log files.


(mike) #8

That’s odd and more than a bit troubling. The only way I can think of that you’d lose report data without deleting the piwik_archive_* tables is if you were generating the reports on demand in the browser. I’m not even sure that would make sense. Sorry I can’t help more. Sounds bad.


(Matthieu Aubry) #9

What is odd is that this appeared in the log files in the archive log after ‘deleting old log files’…'Piwik\Plugins\PrivacyManager\Tasks.deleteReportData,Time elapsed: 0.138s ’ even though ‘report data’ was NEVER set to be deleted ONLY log files.

It’s by design, the task always runs but if you have not enabled the report deletion then it wlil not do anything