Problem? - Archive Data/Stats Display Issues

Support & Bugs
1

I’m new to piwik, but I got it up running and working in good order except for one major problem.

Let me try to summarize my steps:

  1. Installed piwik correctly on my server.

  2. Installed site tracking code correctly on the footer of my site tracking multiple pages.

  3. Checked - stats tracking. Db looks OK (piwik_log_action tracking)

  4. Set up cron job to to automatically archive the reports.
    Here is the following command i used in cpanel:
    0 * * * * /bin/sh /home/usr/site/piwik/misc/cron/archive.sh
    Note: In piwik ‘general settings’ default values used (Allow Piwik archiving to trigger when reports are viewed from the browser = YES; Reports for today will be processed at most every = 10sec)
    Additional Note: I alternatively later disabled auto-triggering since i had the cron job running and put the processing to every 1/2hr), neither setting though would appear to influence the issue experienced.

  5. Cron job reports sent hourly to my site email, here’s an excerpt from the email body for yearly archiving:

Archiving period = year for idsite = 1…

<?xml version="1.0" encoding="utf-8" ?> Archiving for idsite = 1 done! Reports archiving finished. Starting Scheduled tasks...

No data availableFinished Scheduled tasks.

*** In Piwik, there was yearly and weekly archive data available all day today. The cron job is running hourly. Yearly/Weekly Data has been available and showing throughout, also outputted to website OK for several hours.

NUTSHELL: Then all of the data including for ‘today’ gone. Within the hour, ‘today’ data back, archive data/api output gone. Over the course of the last two hours, still no archive data being generated or live visitors, for today = 0.

I heavily suspect that if I go on tomorrow morning, there will be archive data, which will be collected starting fresh, which will then disappear and be deleted later in the day again.

PROBLEM/DETAILS:
Let’s take today for the example where I have been monitoring and setting up an api feed of data onto my site for the duration of the last 8 hours.

Stats have been compiling up fine, where the a) weekly and yearly archives have been showing data and b) the Live Visitors dashboard widget has been both showing data for ‘Today’ and ‘Last 30 minutes.’

Using api I have been outputting data to my site using the following parameters:
piwik/index.php?module=API&method=Actions.getPageUrls&format=XML&idSite=1&period=year&date=yesterday&token_auth=[(enabledtokenhere)]&filter_limit=100

Over the last several hours, everything great, data compiling, api feed working. OK.

Then all the sudden ( A )

  • No data coming through the api feed
  • An xml reading would show this only: and an error message i have triggered is the feed is empty.

( B ) Refreshing Piwik, All archive data disappears, showing “There is no data for this report.” For ‘today’ as well as yearly archive.

A short time later (1/2hr or so, approximately), reports for today reappear minus ‘Live Visitors’ widget: Visits, Pageviews are reset to 0. Last 30 minutes showing hits.

Checking weekly or yearly date range for stats: “There is no data for this report.”

Exporting the XML for instance back from today’s reports that is showing data:
piwik/index.php?module=API&method=Actions.getPageUrls&format=XML&idSite=1&period=day&date=2010-12-06&token_auth=[(enabledtokenhere)]&filter_limit=100

YES, showing data.
Exporting the XML for instance back for yearly or weekly reports of course is outputting nothing except:

The previous couple day’s since I set up piwik I noticed that the stats or archived data from previous day around the same time just disappear.

For my own experience the only thing I’m new at is setting up cron jobs and since I suspect the issue involves something that is obviously happening on a schedule daily, erasing all previous data, I’m guessing it is cron-related?

Could someone please help me figure this out, or let me know what the problem is from my scenario, if my cron command is correct, etc.

Cron command in cpanel again:
0 * * * * /bin/sh /home/usr/site/piwik/misc/cron/archive.sh

THANKS

2

Oh, great… so i read through the other forum posts, specifically:

DATA LOSS
http://forum.piwik.org/read.php?2,66941

The above post if from 2 weeks ago, I’m hoping someone could get back to me -very soon-. I just dropped google analytics for piwik and i hope i’m not going to regret it. :X

Please, if you have input or can help, please see my issue above so we can get this resolved. Honestly, like most forums go, I cannot wait 2 weeks to sort the issue and be able to leave feedback, possible solutions,etc. I saw the archiveprocessing.php files (potential fix? - poster did not leave much info, specifics) and am waiting for someone to advise or confer on my above situation, or whether to try the new?? php file.

Piwik Developers?

3

Well, just try the potential fix there and see if it helps.

4

One thing that you can be confident about is that no data is being deleted or lost in the process. There is a bug regarding timezone calculations for certain timezones setups. We are working on a fix and will release beta release soon, as we haven’t been able to replicate exactly what the problem is.

5

I have tested the issue i was experiencing, replacing the ArchiveProcessing.php file with the one in the “latest.zip” on 11/30/2010 on the main site with the one referred to in matthieu’s post here: http://forum.piwik.org/read.php?2,66941

Again, I don’t know if the ArchiveProcessing.php file was intended to be a fix or is a fix exactly regarding “what,” because i couldn’t find any other info/summary beyond if stating ‘timezone fix, 2 months ago’ (I would of figured that if a major issue was fixed that long ago, the main piwik zip file offered on the homepage would of already been updated with the fix??)

Anyway, replacing this php code or file with the one in the current download package (the one i downloaded and installed 11/30 - again couldn’t find a version number in the package, so i can only offer the date i downloaded) seemed to fix the main problem i was experiencing, which i’ll describe below, although there still is a problem.

***Also, thanks to myself for answering my own question finding out that my cron command was indeed perfectly fine, hopes this helps someone else down the road… and that the problem was only with piwik :wink:

BEFORE SCENARIO 11/30 piwik download package

Example: Stats tracking fine with cron reports emailed to me hourly. 6PM: all archive data (weekly,monthly,yearly) disappears from piwik widgets. An API feed I had hooked up then instead of showing content relayed “No data.” Previously this api was outputting some stats from the yearly archive, on my web stats then did not output. Stats, except ‘for today’ were not visible for the next 6 hours until the “New Day” started at 1AM at which time the archive data would reappear in the widgets and all the stats being flushed to 0 for the new day, would be anew. This cycle would just repeat.
Also note that my cron job for the yearly is posting ‘No Data’ in my email for the job during the ‘piwik display downtime.’

As matthieu stated in response to initial post,

Correct, although it’s essential since piwik’s advantage is the easy availability and export of api data and embeddable widgets that there is no major problem as the current download package offers (11/30/2010). I mean, it’s not really nice to have data being shown on your website or blog then have it disappear throwing an error statement for a 7-hour stretch, is it? Also defeats the whole purpose of having real-time monitoring…some time is better than no-time for 1/3 of the day…

AFTER SCENARIO *replacing the ArchiveProcessing.php file

Stats working, cron jobs ok, showing data being posted. 6PM: All archive data is available in view, AND the api feed data is available and outputting! Major problem solved…BUT under the the stats for today are flushed to 0 in the view and in the ‘Live Visitors!’ widget yet showing stats in the cron job report as I noticed in a 11PM report. Also in the ‘Live Visitors!’ widget, No tracking, remains at 0 until 1PM (when the new day starts). 1PM. All stats begin tracking for the new day, including the ‘today’ Live Visitors counter.

As I mentioned the weekly/monthy/yearly archives and api data are now available during this 7-hour “downtime”:

464

The good and important news is that replacing the ArchiveProcessing.php did fix the “downtime” in the piwik display view and api outputting…although there still is the “downtime” with data not showing for a 7-hour period daily for current day stats.

I would recommend that the download package be updated with the new archiveprocessing file obviously since this is a major fix…(and note possible issues)-- Piwik 1.1 immediately, unless the new version is going to be available in the next few days. If grassroots support is desired, I’m just guessing 75% of the people who would install the package would experience this major issue and ditch piwik and not look back.

Devs/Admins: I would be perfectly willing to help out with communication to keep the most desirable file package available on the site for download w/ with proper documentation. Maybe i’m a stickler, but it seems the situation is a little disorganized on the communication front. I understand how to read and tweak code a bit, but I’m no expert in programming code from scratch, but I certainly feel I could help with documention organization, writing, and keeping track and/or packaging files in addition to assisting developers with their debugging efforts. Feel welcome to ask for assistance if needed. I’m glad the major issue was fixed, I was worried and hoping to be able to stick with piwik. Now I can.

6

Here is the “timezone issue fix” / fixed version ArchiveProcessing.php file I was referring to in my above posts.

dev source page:
http://dev.piwik.org/trac/browser/trunk/core/ArchiveProcessing.php

7

Hello,

We tried to fix this issue in Piwik. We released a 1.1 beta 3 (which will become 1.1 if nothing comes up), and we would really appreciate if you could install it and report if the bug above is still there with this new version.

If you still experience an issue with 1.1b3 version, please create a new forum topic explaining your setup (websites, timezones), what you see and what you expected to see.

To install 1.1b3, it is the same process as a normal upgrade:

Thank you for your help!