Weekly, monthly, yearly reports show zero data for current period right after midnight

Support & Bugs
1

I noticed that since 1.7 weekly, monthly and yearly reports show zero data for the current period in the first hour of the day (after midnight). I think I found out why:

My cron job for archive.php runs 1 minute/31 minutes after the hour. At 00:01, there is sometimes no new visit for the last minute. archive.php reports: “Skipped website id 2, no visit today” and “Archived week/month/year for 0 websites”. For some reason (I’m not quite clear about that), in this situation, any SQL query for monthly/weekly/yearly data looks for rows in piwik_archive_numeric_[currentyear]_[currentmonth] with a ts_archived date value past the last stored ts_archived value and returns empty.

After the next cron run (visits happened), the situation is normal again.

2

Does it mean that during midnight and 00:31, when you access Piwik, you see 0 visits for periods that shouldn’t have?
This would explain the bug experinced by another user in: 301 Moved Permanently

Please confirm or create a ticket with as much info as you can on http://dev.piwik.org/trac

3

matt, yes, 0 visits and 0 or no data for all the other dimensions and metrics.

Upon thinking, I think there are 2 independent problems:

Bug #1: archive.php only considers visits “today” when running: Consider archive.php runs at 00:01 and there haven’t been any visits this day. But there have been visits in the hour before 00:00 (day before). archive.php doesn’t run, because there haven’t been visits “today”, while there are still unprocessed visits after the last run of archive.php (from “day before” )

Bug #2: If archive.php doesn’t run for a website, because there have been no visits since the last run, reports for periods (week/month/year) show no data for current period. Cause for this seems to be, that SQL queries look for a ‘ts_archived’ date value past the last stored ‘ts_archived’ value (I confirmed this only for my case, right after midnight: ts_archived was supposed to be >= “2012-02-28 23:00” while the greatest value for ts_archived for relevant rows in piwik_archive_numeric_2012_02 was “2012-02-28 22:31” )

If you agree with this assessment, I can file 2 independent tickets.

4

Bug #1: archive.php only considers visits “today” when running: Consider archive.php runs at 00:01 and there haven’t been any visits this day. But there have been visits in the hour before 00:00 (day before). archive.php doesn’t run, because there haven’t been visits “today”, while there are still unprocessed visits after the last run of archive.php (from “day before” )

Actually, please update to archive.php from SVN http://dev.piwik.org/svn/trunk/misc/cron/archive.php
Because, in this version, when a website day is finished in its timezone, it should automatically force to trigger all periods processing for the previous day/week/month.
So, this bug should be fixed already, but I would like you to confirm after 24h if that’s the case

Bug #2: If archive.php doesn’t run for a website, because there have been no visits since the last run, reports for periods (week/month/year) show no data for current period. Cause for this seems to be, that SQL queries look for a ‘ts_archived’ date value past the last stored ‘ts_archived’ value (I confirmed this only for my case, right after midnight: ts_archived was supposed to be >= “2012-02-28 23:00” while the greatest value for ts_archived for relevant rows in piwik_archive_numeric_2012_02 was “2012-02-28 22:31” )

Technically, it should also be fixed if you use the latest archive.php from SVN.
Please grab it, and after 24 hours or longer, let me know if you have any issue with it. It is well possible there is some problem still (end of week/month, or other edge cases). PLease report if you have any concern of missing stats!

5

Installed archive.php from SVN, went through nicely so far, all reports there as expected. Had unfortunately 1 visit exactly at midnight, so couldn’t quite put to test - will watch & report if something still not right. Thanks.

6

(I am not trying to hijack the thread, just reporting on what I believe is a similar observation; if I needs to post this in a new thread, please let me know.)

I have been seeing the behavior described above by the original poster as well.

And now in 1.7.1 something else that I believe is related:

  1. I have been running the archive process (I switched to the php version) twice an hour like the OP.
  2. Once a day (at no particular time) Piwik will stop updating the counts for certain sites – in fact, they reset to 0 (including if they previously had had hits counted that day) and won’t update for the rest of the day until the first run after midnight – I effectively have to wait until the first run of the next day to see what their actual count was for the previous day
  3. At one point I increased the frequency to 4 times an hour at 15 minute intervals (and lowered the “Reports for today (or any other Date Range including today) will be processed at most every” to 900 … and that did not make a difference.

(Just an fyi that I am running Piwik in a MySQL replication ring among multiple servers … and it has worked without any problems prior to 1.7.1)

7

piwikeval, try to change archive.php with this version: http://dev.piwik.org/trac/export/5927/trunk/misc/cron/archive.php

Then wait 24 hours and report if you still experience a bug

8

@matt, thank you for your response!

It was solved in a more trivial manner: in my setup I must use “–force-all-websites” in my crontab. No other changes or version upgrades are necessary; it works great now

So, the crontab would look similar to this (for example):

0,30 /usr/bin/php5 /path/to/piwik/archive.php --url=http://www.your-piwik-web-address.tld/ --force-all-websites

9

OK, but still I would appreciate if you could test the new archive.php without --force-all-websites to make sure the bug is really fixed (for all users) Thanks :slight_smile:

10

Well, the “bug” with 0 data for week/month/year is back (new archive.php from SVN), and I’m not sure, but can it be timezone data / DST related? I never noticed before, but the timestamps seem to be GMT, while server time is +01 (CET).

Anyway, week/month/year ran at 23:31 (22:31), were skipped at 00:01 (23:01), reports after 00:00 show no data for periods, and that didn’t happen before (today, I had a server outage and period processing moved from full hour to half hour cron script).

Report from 00:01 (March, 12th) server time CET - timestamp apparently GMT


[2012-03-11 23:01:02] [3f34a1b4] ---------------------------
[2012-03-11 23:01:02] [3f34a1b4] INIT
[2012-03-11 23:01:02] [3f34a1b4] Querying Piwik API at: http://www.[xxx].de/piwik/index.php
[2012-03-11 23:01:02] [3f34a1b4] Running as Super User: admin
[2012-03-11 23:01:02] [3f34a1b4] Notes
[2012-03-11 23:01:02] [3f34a1b4] - Reports for today will be processed at most every 1740 seconds. You can change this value in Piwik UI > Settings > General Settings.
[2012-03-11 23:01:02] [3f34a1b4] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[2012-03-11 23:01:02] [3f34a1b4] - Archiving was last executed without error 29 min 47s ago
[2012-03-11 23:01:02] [3f34a1b4] Will process 1 websites with new visits since 29 min 47s , IDs: 2
[2012-03-11 23:01:02] [3f34a1b4] ---------------------------
[2012-03-11 23:01:02] [3f34a1b4] START
[2012-03-11 23:01:02] [3f34a1b4] Starting Piwik reports archiving...
[2012-03-11 23:01:03] [3f34a1b4] Archived website id = 2, period = day, Time elapsed: 0.887s
[2012-03-11 23:01:03] [3f34a1b4] Archived website id = 2, today = 1 visits, 1 API requests, Time elapsed: 0.887s [1/1 done]
[2012-03-11 23:01:03] [3f34a1b4] Done archiving!
[2012-03-11 23:01:03] [3f34a1b4] ---------------------------
[2012-03-11 23:01:03] [3f34a1b4] SUMMARY
[2012-03-11 23:01:03] [3f34a1b4] Total daily visits archived: 1
[2012-03-11 23:01:03] [3f34a1b4] Archived today's reports for 1 websites
[2012-03-11 23:01:03] [3f34a1b4] Archived week/month/year for 0 websites. 
[2012-03-11 23:01:03] [3f34a1b4] Skipped 0 websites: no new visit since the last script execution
[2012-03-11 23:01:03] [3f34a1b4] Skipped 0 websites day archiving: existing daily reports are less than 1740 seconds old
[2012-03-11 23:01:03] [3f34a1b4] Skipped 1 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[2012-03-11 23:01:03] [3f34a1b4] Total API requests: 1
[2012-03-11 23:01:03] [3f34a1b4] done: 1/1 100%, 1 v, 1 wtoday, 0 wperiods, 1 req, 888 ms, no error
[2012-03-11 23:01:03] [3f34a1b4] Time elapsed: 0.888s
[2012-03-11 23:01:03] [3f34a1b4] ---------------------------
[2012-03-11 23:01:03] [3f34a1b4] SCHEDULED TASKS
[2012-03-11 23:01:03] [3f34a1b4] Starting Scheduled tasks... 
[2012-03-11 23:01:03] [3f34a1b4]  No task to run
[2012-03-11 23:01:03] [3f34a1b4] done

Report from 23:31 (March, 11th) server time


2012-03-11 22:31:01] [c5c438c4] ---------------------------
[2012-03-11 22:31:01] [c5c438c4] INIT
[2012-03-11 22:31:01] [c5c438c4] Querying Piwik API at: http://www.[xxx].de/piwik/index.php
[2012-03-11 22:31:01] [c5c438c4] Running as Super User: admin
[2012-03-11 22:31:01] [c5c438c4] Notes
[2012-03-11 22:31:01] [c5c438c4] - Reports for today will be processed at most every 1740 seconds. You can change this value in Piwik UI > Settings > General Settings.
[2012-03-11 22:31:01] [c5c438c4] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[2012-03-11 22:31:01] [c5c438c4] - Archiving was last executed without error 29 min 57s ago
[2012-03-11 22:31:01] [c5c438c4] Will process 1 websites with new visits since 29 min 57s , IDs: 2
[2012-03-11 22:31:01] [c5c438c4] ---------------------------
[2012-03-11 22:31:01] [c5c438c4] START
[2012-03-11 22:31:01] [c5c438c4] Starting Piwik reports archiving...
[2012-03-11 22:31:03] [c5c438c4] Archived website id = 2, period = day, Time elapsed: 2.021s
[2012-03-11 22:31:08] [c5c438c4] Archived website id = 2, period = week, 10441 visits, Time elapsed: 4.870s
[2012-03-11 22:31:12] [c5c438c4] Archived website id = 2, period = month, 28222 visits, Time elapsed: 3.518s
[2012-03-11 22:31:15] [c5c438c4] Archived website id = 2, period = year, 212595 visits, Time elapsed: 2.935s
[2012-03-11 22:31:15] [c5c438c4] Archived website id = 2, today = 766 visits, 4 API requests, Time elapsed: 13.345s [1/1 done]
[2012-03-11 22:31:15] [c5c438c4] Done archiving!
[2012-03-11 22:31:15] [c5c438c4] ---------------------------
[2012-03-11 22:31:15] [c5c438c4] SUMMARY
[2012-03-11 22:31:15] [c5c438c4] Total daily visits archived: 766
[2012-03-11 22:31:15] [c5c438c4] Archived today's reports for 1 websites
[2012-03-11 22:31:15] [c5c438c4] Archived week/month/year for 1 websites. 
[2012-03-11 22:31:15] [c5c438c4] Skipped 0 websites: no new visit since the last script execution
[2012-03-11 22:31:15] [c5c438c4] Skipped 0 websites day archiving: existing daily reports are less than 1740 seconds old
[2012-03-11 22:31:15] [c5c438c4] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[2012-03-11 22:31:15] [c5c438c4] Total API requests: 4
[2012-03-11 22:31:15] [c5c438c4] done: 1/1 100%, 766 v, 1 wtoday, 1 wperiods, 4 req, 13346 ms, no error
[2012-03-11 22:31:15] [c5c438c4] Time elapsed: 13.346s
[2012-03-11 22:31:15] [c5c438c4] ---------------------------
[2012-03-11 22:31:15] [c5c438c4] SCHEDULED TASKS
[2012-03-11 22:31:15] [c5c438c4] Starting Scheduled tasks... 
[2012-03-11 22:31:15] [c5c438c4]  No task to run
[2012-03-11 22:31:15] [c5c438c4] done

After 00:31 (March 12th) server time (23:31 GMT) period reports reappear normal.


[2012-03-11 23:31:01] [ea0f123b] ---------------------------
[2012-03-11 23:31:01] [ea0f123b] INIT
[2012-03-11 23:31:01] [ea0f123b] Querying Piwik API at: http://www.[xxx].de/piwik/index.php
[2012-03-11 23:31:01] [ea0f123b] Running as Super User: admin
[2012-03-11 23:31:02] [ea0f123b] Notes
[2012-03-11 23:31:02] [ea0f123b] - Reports for today will be processed at most every 1740 seconds. You can change this value in Piwik UI > Settings > General Settings.
[2012-03-11 23:31:02] [ea0f123b] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[2012-03-11 23:31:02] [ea0f123b] - Archiving was last executed without error 29 min 59s ago
[2012-03-11 23:31:02] [ea0f123b] Will process 1 websites with new visits since 29 min 59s , IDs: 2
[2012-03-11 23:31:02] [ea0f123b] ---------------------------
[2012-03-11 23:31:02] [ea0f123b] START
[2012-03-11 23:31:02] [ea0f123b] Starting Piwik reports archiving...
[2012-03-11 23:31:02] [ea0f123b] Archived website id = 2, period = day, Time elapsed: 0.340s
[2012-03-11 23:31:05] [ea0f123b] Archived website id = 2, period = week, 5342 visits, Time elapsed: 3.437s
[2012-03-11 23:31:09] [ea0f123b] Archived website id = 2, period = month, 28243 visits, Time elapsed: 3.635s
[2012-03-11 23:31:12] [ea0f123b] Archived website id = 2, period = year, 212616 visits, Time elapsed: 3.235s
[2012-03-11 23:31:12] [ea0f123b] Archived website id = 2, today = 12 visits, 4 API requests, Time elapsed: 10.647s [1/1 done]
[2012-03-11 23:31:12] [ea0f123b] Done archiving!
[2012-03-11 23:31:12] [ea0f123b] ---------------------------
[2012-03-11 23:31:12] [ea0f123b] SUMMARY
[2012-03-11 23:31:12] [ea0f123b] Total daily visits archived: 12
[2012-03-11 23:31:12] [ea0f123b] Archived today's reports for 1 websites
[2012-03-11 23:31:12] [ea0f123b] Archived week/month/year for 1 websites. 
[2012-03-11 23:31:12] [ea0f123b] Skipped 0 websites: no new visit since the last script execution
[2012-03-11 23:31:12] [ea0f123b] Skipped 0 websites day archiving: existing daily reports are less than 1740 seconds old
[2012-03-11 23:31:12] [ea0f123b] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[2012-03-11 23:31:12] [ea0f123b] Total API requests: 4
[2012-03-11 23:31:12] [ea0f123b] done: 1/1 100%, 12 v, 1 wtoday, 1 wperiods, 4 req, 10648 ms, no error
[2012-03-11 23:31:12] [ea0f123b] Time elapsed: 10.648s
[2012-03-11 23:31:12] [ea0f123b] ---------------------------
[2012-03-11 23:31:12] [ea0f123b] SCHEDULED TASKS
[2012-03-11 23:31:12] [ea0f123b] Starting Scheduled tasks... 
[2012-03-11 23:31:12] [ea0f123b]  No task to run
[2012-03-11 23:31:12] [ea0f123b] done
11

Thanks for the report. I created a ticket at: archive.php: check if website must be reprocessed should be less selective · Issue #3039 · matomo-org/matomo · GitHub
I’m not sure if this bug will fix your problem. please follow up in the ticket directly if you have more info or findings, thanks!

12

A similar bug was fixed in SVN,

Can you please try to use 1.8 RC 5 and report if it is working well ?
released in 1.8 RC5 301 Moved Permanently

13

We definitely fixed all issues related to this bug now :slight_smile:

please update to at least 1.8.3 from there: 301 Moved Permanently

14

Hi, Matt. This issue is happening to me. I opened a ticket but there was no helpful response:

I would like to solve this problem, is there anything you think it could be causing it?