We have noticed a large discrepancy between what our ad/search platforms record as clicks to our site and what Piwik shows as visits. I believe in our case a lot of these missing visits in Piwik are probably due to someone clicking an ad, being redirected through the platforms tracking script and tracked, and before our page is fully loaded, abandoning the page before the Piwik JS code fires.
SO: I was in the process of setting up the PHP Tracking API as an include in our controller, however, I wanted to leave the JS tracking code on the page - was wondering if anyone (devs?) have tried this yet?
Ideally, it would be greate to have the PHP API track necessary information and if / when the JS code fires, gather the less important but nice to have data as well, screen res, etc… Would the JS code update the visit in the log_visit table as its still within the 30 minute window or only add a new action, as the visit is already recorded? I assume this would create a duplicate action?
Ended up changing a couple things around to get this to work for us. I was sort of surprised to see that the PHP API does not set cookies for the visit. Is this an oversight?
I added a method in PiwikTracker.php to place the _pk_id cookie - so the JS code will find the right visitorid. I had to modify the Piwik_Tracker_Visit class a bit as well, in order to update the visitorid with the extra config settings ( screen res and plugins ) that were not detected through PHP. I set the first page title / action, fired from the server side tracking call, as ‘Bounce’.
In recognizeTheVisitor, I check for that action and in handleKnownVisitor, do an update on the log_link_visit_action table to switch it to the correct page view, instead of recording a new action.
The reason for this and the benefit: We now have a more clear view of bounces / abandons. This was missing from our legacy in-house tracking as well, in php when the get request came in for a page, the server side tracking executed and we marked that as an action for the page. However, there was no good way to know if the page actually loaded this way. Simply saying a get request to a page = pageview is definitely flawed. Though, the server side tracking is more accurate to what the ad platforms record as clicks and bill us for and that’s important. We also need to make sure our conversion rates are in line with theirs, especially in platforms where conversion rates effect placement.
Right now, If we get a click we always have initial data tracking as a ‘Bounce’, if the page loads and the JS fires, this is now updated and marked as a proper page view… if not, it really was a bounce/abandon or possibly another problem we need to identify (broken js, etc…) and we can spot trends (UserAgents, IPs, missing plugins, etc…)
Im not psyched about my own execution on this ( I needed a quick way to fix the variance between our reporting and the ad platform’s) - but I think there is definitely benefit here. I need to find a better way to check if the action should be updated from Bounce, beside relying on the idaction which could change.
Reading _pk_id cookie is definitely on the TODO list if it is not working yet. Please submit your improvements to the code (in separate patches for separate changes if possible) in a ticket in trac and we will gladly commit them after review
Hey kmfk, it seems you do a great job. Right now I’m trying to figure out how to implement almost the same things. Is any way you share the files you updated during the customization of your Piwik installation? I would be greatly appreciated for your help! Thank you.
Yeah, I dont mind sharing some of the stuff I’ve modified. I’m still planning on submitting some of this stuff, once I get a ticket(s) in, I’ll link to them here. Like I said above, there were some changes I wanted to make first as a lot of the things I changed were pretty specific to our case and wouldn’t really work for most people. I’ll follow up shortly.
You should let me know what things you’re looking to change and implement and I’ll also see what other examples I can give you.
Thank you for a quick response kmfk. I have a similar problem that there is a discrepancy between our ad-server stats and piwik js stats. That’s why I need to combine server-side tracking with client-side one. My goal is to track visitors on php but still save all the extended client-side statistics. I found you’ve made exactly all the necessary things that I should do to get it. I like your idea to mark a view as ‘bounced’ and then update the status once js is loaded + more client statistics. Also you made an important fix to place _pk_id cookie on server-side. So if you could just provide these changes as a patch or so it would be awesome. I have the latest version of Piwik.
Take a look at #2699 - The patch is listed there. That was based off the newest revisions from trunk, but you should be able to add the methods to your own files.
Let me know if that works out for you.
edit: I mentioned it in the ticket comments, but I did have to set the following config option:
[Tracker]
trust_visitors_cookies = 1
It was implemented for intranets - but because config_id hash will almost always be different between PHP and JS, this was necessary
kmfk, great thank you for your help! I finally implemented the features I needed. After reviewing your code I found 2 issues:
When it updates idaction_name and idaction_url in Visit.php (changes from ‘bounce’ to actual values) it does update ALL actions since the user visited a site. So looks like he visited the same page many times during his session. I changed the code a little bit:
//Update 'Abandon' action for current idvisit - Only happens on Visit's first Action ( avoids duplicating action when tracking Abandons)
if ($this->trackAbandons)
{
$sql = "select MAX(idlink_va) 'idlink_va' from ". Piwik_Common::prefixTable('log_link_visit_action') ." where idvisit = ? LIMIT 1";
$visitActionRow = Piwik_Tracker::getDatabase()->fetch($sql, array($this->visitorInfo['idvisit']));
if($visitActionRow && count($visitActionRow) > 0)
{
$updateActionQuery = "UPDATE ". Piwik_Common::prefixTable('log_link_visit_action') ." SET idaction_name = ?, idaction_url = ? ".
"WHERE idlink_va = ?";
$updatedAction = Piwik_Tracker::getDatabase()->query($updateActionQuery, array( $idActionName, $idActionUrl, $visitActionRow['idlink_va'] ) );
}
}
When you track via PHP API you should set a visitor ID based on previous user’s cookie:
$piwik->setVisitorId($piwik->getVisitorId());
I spent many hours debuging piwik’s source until found that out
Hope it helps somebody who will use this code too.
Definitely oversight on my part. Our target sites have only one php page and a multi-part javasript lead capture form. Our actions are actually the different steps of the form, which we update through ajax. So for us, we only needed to fire the PHP tracking when the user first visited. Does make sense to force the visitorId from the cookie in cases of reloads and such.
The only change I would make is to save the extra query and just update the action based on server_time . Matter of fact, the idaction_url shouldn’t be changing at all, so we could do this:
$updateActionQuery = "UPDATE ". Piwik_Common::prefixTable('log_link_visit_action') ." SET idaction_name = ? ".
WHERE idvisit = ? AND idaction_url = ? ".
"ORDER BY server_time ASC LIMIT 1";
$updatedAction = Piwik_Tracker::getDatabase()->query($updateActionQuery, array( $idActionName, $idActionUrl, $this->visitorInfo['idvisit'] ) );
I really only intended to look at the first action (visit), not each action during their session, to make sure we had the correct number of visits.
