Historical data recorded through tracking API does not show up in GUI


#1

Alas, having searched this forum and tried the suggestions others have posed to similar issues, I have run into an issue in my attempts to use the tracking API to import historical data into Piwik—Having used the tracking API (https://piwik.org/docs/tracking-api/reference/#toc-tracking-api-reference), I have noticed that although the data appears in the database, it does not show up in the user interface. I DID try to force Piwik to re-run its reports, and here are the steps I took. (I checked the GUI after each step to see if the problem was solved, but unfortunately, it was not.) Can you please let me know what I might be doing wrong?

  1. Ensured that “Delete old logs” was OFF in the Settings area.
  2. Deleted the archive tables from the database.
  3. Made requests to the tracking API, using the param
    &method=CoreAdminHome.invalidateArchivedReports to let Piwik know that the historical data I was
    sending it was after the fact. (Tried also without this parameter, too.)
  4. Ran the archive.php cron job.

Here’s a sample array of data typical of what I sent to the API:

Array
(
[idsite] => 47
[rec] => 1
[ url ] => sitetobetracked.com
[action_name] => Home
[rand] => 292
[apiv] => 1
[_idvc] => 1
[_viewts] => 1374451200
[_idts] => 1374451200
[res] => 1366x768
[ua] => Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/31.0.1650.57 Safari/537.36
[lang] => en-us,en
[cip] => 11.22.33.44
[cdt] => 2013-07-22
[new_visit] => 1
[country] => us
[region] => MI
)


(Matthieu Aubry) #2

input request is missing the token_auth required for the &cdt parameter


#3

Hi Matt, actually the request does have the token_auth…I just double checked. It’s just not printed there.

Here’s an actual request URL, with real values removed, to avoid confusion:
http://path/to/piwik/piwik.php?idsite=38&rec=1&url=http%3A%2F%2Fsitetobetracked.com&action_name=Home&rand=12&apiv=1&_idvc=1&res=1366x768&ua=Mozilla%2F5.0+(Windows+NT+6.1%3B+WOW64%3B+rv%3A13.0)+Gecko%2F20100101+Firefox%2F13.0.1&lang=en-us%2Cen&cip=11.22.33.44&cdt=2013-10-13&new_visit=1&country=us&region=MI&format=JSON&module=API&method=CoreAdminHome.invalidateArchivedReports&token_auth=xxxx

I seem to get a successful response from the API, so also for that reason, I don’t think there’s an issue with the token_auth. The API request seems to be successful, and I see all of the info show up in the database. What doesn’t make sense to me is why none of that data shows up on the GUI. I DID try to force Piwik to re-run its reports, and here are the steps I took. (I checked the GUI after each step to see if the problem was solved, but unfortunately, it was not.) Can you please let me know what I might be doing wrong?

  1. Ensured that “Delete old logs” was OFF in the Settings area.
  2. Deleted the archive tables from the database.
  3. Made requests to the tracking API, using the param
    &method=CoreAdminHome.invalidateArchivedReports to let Piwik know that the historical data I was
    sending it was after the fact. (Tried also without this parameter, too.)
  4. Ran the archive.php cron job.

Additionally, running archive.php suggests that Piwik doesn’t recognize the historical data at all…even though that data came in through its own API. Here’s a sample output from archive.php:

[2013-12-10 17:48:51] [xxxxxxxx] [3.34 Mb] Archived website id = 38, period = day, Time elapsed: 0.089s
[2013-12-10 17:48:52] [xxxxxxxx] [3.48 Mb] Archived website id = 38, period = week, 0 visits, Time elapsed: 0.729s
[2013-12-10 17:48:52] [xxxxxxxx] [3.35 Mb] Archived website id = 38, period = month, 0 visits, Time elapsed: 0.237s
[2013-12-10 17:48:52] [xxxxxxxx] [3.34 Mb] Archived website id = 38, period = year, 0 visits, Time elapsed: 0.166s
[2013-12-10 17:48:52] [xxxxxxxx] [3.33 Mb] Archived website id = 38, today = 0 visits, 4 API requests, Time elapsed: 1.223s [17/17 done]

Maybe there is another way to go about re-running reports? Perhaps the above process isn’t enough? Any insight on this would be greatly appreciated.


(Matthieu Aubry) #4

are you using 2.0 beta ? New Piwik 2.0 public beta for testers - Analytics Platform - Matomo

If so I’d check in log_visit or Visitor log that the dates rare recorded in the past

If they are then check your API call to invalidateArchivedReports
it should return a success message


#5

Thanks for your help so far!

We’re using Piwik 1.2.

I believe I’m getting successful responses from the API when I use CoreAdminHome.invalidateArchivedReports (Sample URL with params is http://path/to/piwik/piwik.php?idsite=38&rec=1&url=http%3A%2F%2Fsitetobetracked.com&action_name=Home&rand=12&apiv=1&_idvc=1&res=1366x768&ua=Mozilla%2F5.0+(Windows+NT+6.1%3B+WOW64%3B+rv%3A13.0)+Gecko%2F20100101+Firefox%2F13.0.1&lang=en-us%2Cen&cip=11.22.33.44&cdt=2013-10-13&new_visit=1&country=us&region=MI&format=JSON&module=API&method=CoreAdminHome.invalidateArchivedReports&token_auth=xxxx )

Here are the responses I get…

WITHOUT the debugging on:
string(43) “GIF89a�!�,D;”

WITH debugging on:
string(5557) "Debug enabled - Input parameters:
array ( ‘idsite’ => ‘38’, ‘rec’ => ‘1’, ‘url’ => ‘http://sitetobetracked.com’, ‘action_name’ => ‘Home’, ‘rand’ => ‘659’, ‘apiv’ => ‘1’, ‘_idvc’ => ‘1’, ‘res’ => ‘1366x768’, ‘ua’ => ‘Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1’, ‘lang’ => ‘en-us,en’, ‘cip’ => ‘’, ‘cdt’ => ‘2013-10-13’, ‘new_visit’ => ‘1’, ‘country’ => ‘US’, ‘region’ => ‘MI’, ‘format’ => ‘JSON’, ‘module’ => ‘API’, ‘method’ => ‘CoreAdminHome.invalidateArchivedReports’, ‘token_auth’ => ‘xxxx’, )
token_auth is authenticated!
Loading plugins: { Provider,Goals,DoNotTrack,UserCountry }
Current datetime: 2013-10-13 00:00:00
(this is not a Site Search request)
Action is a Page URL, Action name = Home, Action URL = http://annarborartsandcrafts.com
Piwik_Cookie::__set_state(array(
‘name’ => ‘_pk_uid’,
‘expire’ => 1444694400,
‘path’ => ‘’,
‘domain’ => ‘’,
‘secure’ => false,
‘httponly’ => false,
‘value’ =>
array (
),
‘keyStore’ => false,
))
Visitor doesn’t have the piwik cookie…
The visitor was not matched with an existing visitor…
Visitor detected, but last action was more than 30 minutes ago…
New Visit (IP = 11.22.33.44)
GEO: no current location provider sent, falling back to default ‘default’ one.
GEO: Found IP location (provider ‘default’): array ( ‘country_code’ => ‘us’, ‘continent_code’ => ‘amn’, ‘continent_name’ => ‘UserCountry_continent_amn’, ‘country_name’ => ‘UserCountry_country_us’, )
array (
‘idsite’ => 38,
‘visitor_localtime’ => ‘00:00:00’,
‘idvisitor’ => ‘xxxx’,
‘visitor_returning’ => 0,
‘visitor_count_visits’ => 1,
‘visitor_days_since_last’ => 0,
‘visitor_days_since_order’ => 0,
‘visitor_days_since_first’ => 0,
‘visit_first_action_time’ => ‘2013-10-13 00:00:00’,
‘visit_last_action_time’ => ‘2013-10-13 00:00:00’,
‘visit_entry_idaction_url’ => 2405,
‘visit_entry_idaction_name’ => 2404,
‘visit_exit_idaction_url’ => 2405,
‘visit_exit_idaction_name’ => 2404,
‘visit_total_actions’ => 1,
‘visit_total_searches’ => 0,
‘visit_total_time’ => 0,
‘visit_goal_converted’ => 0,
‘visit_goal_buyer’ => 0,
‘referer_type’ => 1,
‘referer_name’ => ‘’,
‘referer_url’ => ‘’,
‘referer_keyword’ => ‘’,
‘config_id’ => ‘xxxx’,
‘config_os’ => ‘WI7’,
‘config_browser_name’ => ‘FF’,
‘config_browser_version’ => ‘13.0’,
‘config_resolution’ => ‘1366x768’,
‘config_pdf’ => 0,
‘config_flash’ => 0,
‘config_java’ => 0,
‘config_director’ => 0,
‘config_quicktime’ => 0,
‘config_realplayer’ => 0,
‘config_windowsmedia’ => 0,
‘config_gears’ => 0,
‘config_silverlight’ => 0,
‘config_cookie’ => 0,
‘location_ip’ => ‘xxxx’,
‘location_browser_lang’ => ‘en-us,en’,
‘location_country’ => ‘us’,
‘location_region’ => ‘MI’,
‘location_provider’ => ‘verizon.net’,
)
array (
‘idvisit’ => 1127,
‘idsite’ => 38,
‘idvisitor’ => ‘xxxx’,
‘server_time’ => ‘2013-10-13 00:00:00’,
‘idaction_url’ => ‘2405’,
‘idaction_name’ => 2404,
‘idaction_url_ref’ => 0,
‘idaction_name_ref’ => 0,
‘time_spent_ref_action’ => 0,
)
Piwik_Cookie::__set_state(array(
‘name’ => ‘_pk_uid’,
‘expire’ => 1444694400,
‘path’ => ‘’,
‘domain’ => ‘’,
‘secure’ => false,
‘httponly’ => false,
‘value’ =>
array (
),
‘keyStore’ => false,
))
-> Scheduled tasks not triggered.
Next run will be from: 2013-12-11 18:31:37 UTC
Nothing to notice => default behaviour
End of the page.
array (
)
Piwik_Timer::__set_state(array(
‘timerStart’ => 1386784910.6297,
‘memoryStart’ => 1078744,
))


(Matthieu Aubry) #6

please update to 2.0 beta as we fixed so many bugs and made many changes to all this code! New Piwik 2.0 public beta for testers - Analytics Platform - Matomo


#7

Hey Matt, I just upgraded to the beta version–and that totally solved it! So, anyone else having issue with historical data…I second Matt’s suggestion…solved everything & imported historical data is showing up now :slight_smile:

Steps I took:

Settings > General Settings > select “The latest beta release” for version to use

Check for updates in the About Piwik box on the home dashboard

Use automatic update option. A white screen will appear. Click “back” in the browser.

Choose Continue to Piwik…

The screen for “Database update required” appears. It says to something via command line; I did so.

Ensure that “Delete old logs” is OFF in the Settings area

Delete the archive tables from the database


#8

OK, one more oddism.

The procedure I described as successful was completed on a dev server. Upon trying to duplicate this on the live server, I couldn’t even get Piwik to update. I did the following, but the option to update didn’t appear on the live server the way it did on the dev:

  1. Settings > General Settings > select “The latest beta release” for version to use
  2. Check for updates in the About Piwik box on the home dashboard

Any ideas why Piwik 1.2 wouldn’t recognize that there’s an update available, after following the steps above?


(Matthieu Aubry) #9

Because your server is maybe not running at least php 5.3.2 ?


#10

That server is running 5.3.26


#11

I’m wondering if anyone else has any suggestions. I followed the exact same process on the live server that worked successfully on the dev server. However, the live server does NOT recognize that a new version of Piwik is available. This is absolutely to our data migration, so any help here would be greatly appreciated!


#12

someone else had a problem and they were running 5.2.4 php yours is close they updated to 5.3.3 and all their issues went away… possible cor you to?


#13

I have exactly the same problem! PHP 5.3 + and latest MySQL on Ubuntu 12.04


(system) #14

http://www.reddit.com/r/drseuss/comments/1zelsj

http://www.reddit.com/r/drseuss/comments/1zelsj

http://www.reddit.com/r/drseuss/comments/1zelsj

http://www.reddit.com/r/drseuss/comments/1zelsj