Generate Random Data for Piwik


#1

Hello

I have a script which I run from the command line (http://pastebin.com/dj6V38wt​) which uses the Tracking API to generate random data and send requests to piwik.php. When I run this script without my token_auth, I can see the visits and page views being tracked in Piwik. When I run this script with my token_auth the data is accepted by Piwik but not visible in the user interface. (See attachments).

Sadly, I can’t seem to force Piwik to show (and compute the reports) for data older than the current date. I have been looking quite extensively over Google and StackOverflow and this was the best I can find: ​http://stackoverflow.com/questions/17920198/piwik-import-historical-data-via-the-api-shows-only-last-two-days-in-dashboard


These are two responses for the HTTP requests made by my script to piwik.php using the Tracking API. One with and one without the token_auth.

Without Token Auth


Debug enabled - Input parameters: <br/>array ( 'idsite' => '1', 'rec' => '1', 'apiv' => '1', 'r' => '752987', 'cip' => '60.34.172.193', 'cid' => '288a630c8a3e7445', 'cdt' => '2011-04-14 10:13:55', '_idts' => '1392570826', '_idvc' => '0', 'fla' => '0', 'java' => '0', 'dir' => '0', 'qt' => '0', 'realp' => '0', 'pdf' => '0', 'wma' => '0', 'gears' => '0', 'ag' => '0', 'h' => '7', 'm' => '23', 's' => '5', 'res' => '320x480', 'cookie' => '1', 'url' => 'http://www.ncl.ac.uk/about/quality/research.htm', 'urlref' => 'http://www.ncl.ac.uk/research/', 'action_name' => 'Research Excellence - About - Newcastle University', )
Loading plugins: { Provider,Goals,PrivacyManager,UserCountry,UsersManager,SitesManager }
Current datetime: 2014-02-16 17:18:56
Visitor IP was _not_ anonymized: 10.0.2.2
(this is not a Site Search request)
Action is a PAGE_URL, Action name = Research Excellence - About - Newcastle University, Action URL = http://www.ncl.ac.uk/about/quality/research.htm
Visitor doesn't have the piwik cookie...
The visitor is known (idvisitor = c21f1bc603ed84cd, config_id = cb000919b43db239, idvisit = 43, last action = Sun, 16 Feb 2014 17:18:56 +0000, first action = Sun, 16 Feb 2014 17:14:04 +0000, visit_goal_buyer' = 0)
Visit is known (IP = 10.0.2.2)
Updating existing visit: array ( 'visit_exit_idaction_name' => '85', 'visit_exit_idaction_url' => 86, 'visit_total_actions' => 'visit_total_actions + 1', 'visit_last_action_time' => '2014-02-16 17:18:56', 'visit_total_time' => 293, 'idvisitor' => 'c21f1bc603ed84cd', 'visit_goal_buyer' => '0', )
Inserted new action:
array (
  'idvisit' => '43',
  'idsite' => 1,
  'idvisitor' => 'ÂeÆí„Í',
  'server_time' => '2014-02-16 17:18:56',
  'idaction_url' => '86',
  'idaction_name' => '85',
  'idaction_url_ref' => '86',
  'idaction_name_ref' => '85',
  'time_spent_ref_action' => 0,
  'idlink_va' => '682',
)
-> Scheduled tasks not running in Tracker: Browser archiving is disabled.
Nothing to notice => default behaviour
End of the page.
array (
  'piwik_auth' => 'login=czo1OiJhZG1pbiI7:token_auth=czozMjoiOTFmYjliZTFjZjQ4YTdjYjc5NjIxNWJkYTA5OThmOGUiOw==:_=eeb76d23a72de37b8b631408c3e863dbc3881a6f',
  'PIWIK_SESSID' => 'vhgn8achn7cj5kj9k28agjusa0',
)
Piwik\Timer::__set_state(array( 'timerStart' => 1392571136.6637, 'memoryStart' => 3018592, ))

With Token Auth


Debug enabled - Input parameters: <br/>array ( 'idsite' => '1', 'rec' => '1', 'apiv' => '1', 'r' => '525016', 'cip' => '15.18.171.171', 'cid' => '0ce3593df1c7748e', 'cdt' => '2010-05-01 16:32:10', 'token_auth' => 'cd27e77092ee3f700785f450ca7785ac', '_idts' => '1392570895', '_idvc' => '0', 'fla' => '0', 'java' => '0', 'dir' => '0', 'qt' => '0', 'realp' => '0', 'pdf' => '0', 'wma' => '0', 'gears' => '0', 'ag' => '0', 'h' => '22', 'm' => '54', 's' => '21', 'res' => '768x1024', 'cookie' => '1', 'url' => 'http://www.ncl.ac.uk/undergraduate/degrees/cc18/courseoverview/', 'urlref' => 'http://en.wikipedia.org/wiki/Newcastle_University', 'action_name' => 'Biology and Psychology BSc Honours : Course Overview - Undergraduate Study - Newcastle University', )
token_auth is authenticated!
Loading plugins: { Provider,Goals,PrivacyManager,UserCountry,UsersManager,SitesManager }
Current datetime: 2014-02-16 17:18:54
Visitor IP was _not_ anonymized: 15.18.171.171
(this is not a Site Search request)
Action is a PAGE_URL, Action name = Biology and Psychology BSc Honours : Course Overview - Undergraduate Study - Newcastle University, Action URL = http://www.ncl.ac.uk/undergraduate/degrees/cc18/courseoverview/
Request will be recorded for this idvisitor = 0ce3593df1c7748e
Matching visitors with: visitorId=0ce3593df1c7748e OR configId=de5ae69a12fae43d
The visitor is known (idvisitor = 0ce3593df1c7748e, config_id = de5ae69a12fae43d, idvisit = 46, last action = Sat, 01 May 2010 16:32:10 +0000, first action = Sat, 01 May 2010 16:31:41 +0000, visit_goal_buyer' = 0)
Visit is known (IP = 15.18.171.171)
Updating existing visit: array ( 'visit_exit_idaction_name' => '23', 'visit_exit_idaction_url' => 24, 'visit_total_actions' => 'visit_total_actions + 1', 'visit_last_action_time' => '2010-05-01 16:32:10', 'visit_total_time' => 30, 'idvisitor' => '0ce3593df1c7748e', 'visit_goal_buyer' => '0', )
Inserted new action:
array (
  'idvisit' => '46',
  'idsite' => 1,
  'idvisitor' => ' ãY=ñÇtŽ',
  'server_time' => '2010-05-01 16:32:10',
  'idaction_url' => '24',
  'idaction_name' => '23',
  'idaction_url_ref' => '24',
  'idaction_name_ref' => '23',
  'time_spent_ref_action' => 0,
  'idlink_va' => '678',
)
Nothing to notice => default behaviour
End of the page.
array (
  'piwik_auth' => 'login=czo1OiJhZG1pbiI7:token_auth=czozMjoiOTFmYjliZTFjZjQ4YTdjYjc5NjIxNWJkYTA5OThmOGUiOw==:_=eeb76d23a72de37b8b631408c3e863dbc3881a6f',
  'PIWIK_SESSID' => 'vhgn8achn7cj5kj9k28agjusa0',
)
Piwik\Timer::__set_state(array( 'timerStart' => 1392571134.1059, 'memoryStart' => 3020124, ))​

Why would the data for the second request with the token_auth not show in Piwik dashboard, but the first would?

Many thanks!


(Matthieu Aubry) #2

see: How do I record tracking data in the past, and tell Piwik to invalidate and re-process the past reports? - Analytics Platform - Matomo


#3

i believe that providing the token_auth switches the behavior to expect bulk formatted input data (json), whereas without it, it expects single urls

http://developer.piwik.org/api-reference/tracking-api


#4

[quote=matt]
see: http://piwik.org/faq/how-to/faq_155/[/quote]

Is it possible to do this for a range of dates, rather than having to specify every date over the last x years?


#5

I am using the following command to force archiving for the last 6 years, but Piwik doesn’t seem to actually consider any data before 01-01-2014?


php archive.php --url=http://localhost/piwik --token-auth=xxx --force-all-periods=203212800


INFO [2014-03-02 19:18:10] [c00fc] ---------------------------
INFO [2014-03-02 19:18:10] [c00fc] INIT
INFO [2014-03-02 19:18:10] [c00fc] Querying Piwik API at: http://localhost/piwik/index.php
INFO [2014-03-02 19:18:10] [c00fc] Running Piwik 2.0.3 as Super User: admin
INFO [2014-03-02 19:18:10] [c00fc] ---------------------------
INFO [2014-03-02 19:18:10] [c00fc] NOTES
INFO [2014-03-02 19:18:10] [c00fc] - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Piwik UI > Settings > General Settings.
INFO [2014-03-02 19:18:10] [c00fc]   See the doc at: http://piwik.org/docs/setup-auto-archiving/
INFO [2014-03-02 19:18:10] [c00fc] - Reports for today will be processed at most every 10 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO [2014-03-02 19:18:10] [c00fc] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO [2014-03-02 19:18:10] [c00fc] - Archiving was last executed without error 11s ago
INFO [2014-03-02 19:18:10] [c00fc] - Will process 1 websites with new visits since 6 years 160 days , IDs: 1
INFO [2014-03-02 19:18:10] [c00fc] ---------------------------
INFO [2014-03-02 19:18:10] [c00fc] START
INFO [2014-03-02 19:18:11] [c00fc] Archived website id = 1, period = day, Time elapsed: 0.796s
INFO [2014-03-02 19:18:12] [c00fc] Archived website id = 1, period = week, 659 visits, Time elapsed: 1.792s
INFO [2014-03-02 19:18:14] [c00fc] Archived website id = 1, period = month, 395 visits, Time elapsed: 1.360s
INFO [2014-03-02 19:18:15] [c00fc] Archived website id = 1, period = year, 395 visits, Time elapsed: 1.166s
INFO [2014-03-02 19:18:15] [c00fc] Archived website id = 1, today = 95 visits, 4 API requests, Time elapsed: 5.117s [1/1 done]
INFO [2014-03-02 19:18:15] [c00fc] Starting Piwik reports archiving...
INFO [2014-03-02 19:18:15] [c00fc] Done archiving!
INFO [2014-03-02 19:18:15] [c00fc] ---------------------------
INFO [2014-03-02 19:18:15] [c00fc] SUMMARY
INFO [2014-03-02 19:18:15] [c00fc] Total daily visits archived: 95
INFO [2014-03-02 19:18:15] [c00fc] Archived today's reports for 1 websites
INFO [2014-03-02 19:18:15] [c00fc] Archived week/month/year for 1 websites
INFO [2014-03-02 19:18:15] [c00fc] Skipped 0 websites: no new visit since the last script execution
INFO [2014-03-02 19:18:15] [c00fc] Skipped 0 websites day archiving: existing daily reports are less than 10 seconds old
INFO [2014-03-02 19:18:15] [c00fc] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO [2014-03-02 19:18:15] [c00fc] Total API requests: 4
INFO [2014-03-02 19:18:15] [c00fc] done: 1/1 100%, 95 v, 1 wtoday, 1 wperiods, 4 req, 5118 ms, no error
INFO [2014-03-02 19:18:15] [c00fc] Time elapsed: 5.118s
INFO [2014-03-02 19:18:15] [c00fc] ---------------------------
INFO [2014-03-02 19:18:15] [c00fc] SCHEDULED TASKS
INFO [2014-03-02 19:18:15] [c00fc] Starting Scheduled tasks...
INFO [2014-03-02 19:18:15] [c00fc]  No task to run
INFO [2014-03-02 19:18:15] [c00fc] done
INFO [2014-03-02 19:18:15] [c00fc] ---------------------------