Archiving yearly stats seems to break Piwik

Alocer · March 17, 2015, 3:50pm

Hi,

We've had issues on one of our Piwik installs. Whenever we try to archive data for all periods using a cron job, it shows data for every period as "0" or "There is no data for this report" on the Piwik GUI for any period.
We then started archiving using cron jobs to archive specific periods. Everything works for days, weeks, and months now, but it still behaves as previously mentioned when it comes to years. From there we can assume that it is during the process of archiving the yearly data that something breaks.

-edit for the sake of clarity -
What I mean is that since we are archiving specific periods, these stats show up correctly. However archiving the stats for this year as a specific period still shows No Data for this report, but only for years (it doesn't break other periods).
- end of edit -

There is no error message in our logs when trying to archive for either yearly or all periods (see below). I've dug through our logs looking for a reason this would happen but could not find anything.

Thanks!

Log output for all periods (ran this weekend, all stats were at 0 in the GUI after this):

INFO CoreConsole[2015-03-14 15:30:03] [8bec1] ---------------------------
INFO CoreConsole[2015-03-14 15:30:03] [8bec1] INIT
INFO CoreConsole[2015-03-14 15:30:03] [8bec1] Piwik is installed at: https://piwik.mysite.org/index.php
INFO CoreConsole[2015-03-14 15:30:03] [8bec1] Running Piwik 2.8.3 as Super User
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] ---------------------------
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] NOTES
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] - Reports for today will be processed at most every 1800 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] - Archiving was last executed without error 11 hours 0 min ago
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] - Will process 1 websites with new visits since 11 hours 0 min , IDs: 1
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] ---------------------------
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] START
INFO CoreConsole[2015-03-14 15:30:04] [8bec1] Starting Piwik reports archiving...
INFO CoreConsole[2015-03-14 15:32:07] [8bec1] Archived website id = 1, period = day, 34461 visits in last last2 days, 3586 visits today, Time elapsed: 122.224s
INFO CoreConsole[2015-03-14 15:52:31] [8bec1] Archived website id = 1, period = week, 3916546 visits in last last140 weeks, 190622 visits this week, Time elapsed: 1224.332s
INFO CoreConsole[2015-03-14 17:56:40] [8bec1] Archived website id = 1, period = month, 3916364 visits in last last52 months, 425590 visits this month, Time elapsed: 7449.652s
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Archived website id = 1, period = year, 3916364 visits in last last7 years, 2072745 visits this year, Time elapsed: 21130.107s
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Archived website id = 1, 4 API requests, Time elapsed: 29926.317s [1/1 done]
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Done archiving!
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] ---------------------------
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] SUMMARY
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Total visits for today across archived websites: 3586
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Archived today's reports for 1 websites
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Archived week/month/year for 1 websites
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Skipped 0 websites day archiving: existing daily reports are less than 1800 seconds old
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Total API requests: 4
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] done: 1/1 100%, 3586 vtoday, 1 wtoday, 1 wperiods, 4 req, 29926475 ms, no error
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Time elapsed: 29926.475s
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] ---------------------------
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] SCHEDULED TASKS
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] Starting Scheduled tasks...
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] task,output
Piwik\Plugins\ExamplePlugin\Tasks.myTask,Time elapsed: 0.000s
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] done
INFO CoreConsole[2015-03-14 23:48:51] [8bec1] ---------------------------

Alocer · March 17, 2015, 5:47pm

I have also just noticed that there are no entries in my database where period = 4 in piwik_archive_numeric_2015_01.
This would probably explain why there doesn’t seem to be any yearly stats in the Piwik GUI. However, we did have stats present there at some point (we were able to manually run the archive command from command line at one point). This means that at some point, the entries in the database would have gotten erased?
What would be the course of action to follow at this point? Do I have to regenerate the stats for all months of 2015? If so, would the fact that the option to “Delete old visitor logs and reports” is set to 60 days be a problem?

matthieu · March 19, 2015, 9:45am

Hi there,

Please try using 2.12.0 RC which makes archiving much more efficient - maybe this solve your issue? I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo

Alocer · March 19, 2015, 2:22pm

Hi Matt,

Thanks for the reply!
We are currently in the process of evaluating a time period where it would be convenient for us to upgrade our Piwik installation. Are the effiency modifications to archiving only in 2.12.0 RC? Being that our Piwik installs are in a production environment we would rather wait for a stable version to come out before upgrading. Is a release expected soon?

Regarding my second post, would I be right to assume that since our Piwik deletes old logs that are 60 days or older, regenerating the stats for January would be impossible? In the web GUI, the informations regarding old log deletion mentions that “you must ensure that all previous daily reports have been processed, so that no data is lost”. Since our daily data has always been processed, would we be good to go?

Since there are currently stats present for day/week/month in the database, would Piwik be able to rebuild the necessary yearly data from those, or are yearly reports built from logs (that like I said, we might no longer have for periods older than 60 days)? A clarification on how the archive process generates the yearly reports and its interaction with old log deletions would be very useful!

Thanks!