Paypal IPN goal tracking and keywords


#1

Hello all, I’m new to Piwik and loving it so far. I started using it due to Google Analytics complete inability to integrate with paypal (their “solution” only works for about 20% of visitors). I got Piwik installed quickly and without trouble, and to my pleasant surprise it was really easy getting e-commerce transactions to register as goals, with the correct revenue values, from my paypal IPN script using the advanced php tracking method. For those who aren’t aware, javascript will not be executed in the IPN script.

Here’s the rub: to have real value, I need to associate these goals with the keyword/campaign/referrer that sent the visitors to me in the first place. Here’s what I’m thinking should be possible. From my website, just before the user clicks the button to go pay on paypal’s website, I could grab some sort of value (visitorID?) and pop it into a custom variable that paypal would return to me when it calls my IPN script. I could then send that back to Piwik when I trigger the goal via the PiwikTracker.php script. I’m just not certain what value I should grab to passthru to my IPN script, and then what I should add to the PiwikTracker call to make the association between original visitor and the goal.

I’ve searched the forum carefully before posting this - quite surprised that nobody else has raised this issue before. Hopefully this should be an easy one for some of you Piwik guru’s out there. Any help will be very much appreciated!

UPDATE: IPN Paypal tracking How to published!!


(Matthieu Aubry) #2

morrack,
What is the solution that GA proposes that works 20% of the time?

Piwik attributes conversions to a Keyword or website or campaign, based on the value in the First party cookie “_pk_ref.*****”. This cookie is set with the referer used to reach the site.
Problem is when you trigger conversion on paypal, you don’t have the 1st party cookies available.

Solutions?

  1. trigger the Goal on your website so that this cookie is passed succesfuly to Piwik. (I guess you trigger it on Paypal to ensure that all conversions are tracked, in case users close the paypal site before coming back to your site?)

  2. we could pass the Referers (found in the cookie) to Paypal (somehow? not sure what is possible there), and then we could pass the referer to the PHP tracking image. I don’t think we can do that currently, but that should be easy to add.

Let me know. In any case, we would really want to help you here and make sure Piwik works in this use case, because as you point out, it’s a very important use case.


#3

Hey Matt, I really appreciate your willingness to make this happen!

Paypal in a nutshell:
1 - order page on my site with a paypal button (html form). I can embed custom variables into this form that paypal will pass back to my IPN script, so if I can obtain the visitor ID at this point and turn it into a variable, I can get that info back at the critical point.
2 - On submitting the form, the user is redirected to paypal.com to complete the sale
3 - User see’s a thank you page on paypal.com, and may or may not (80% of the time will not) then carry on to end up on a thank you page back at my site. GA’s “solution” is to trigger the goal on this thank you page on my site. That’s why I say it only works 20% of the time.
4 - Sometime after this (usually within a few minutes) paypal will call my IPN script, also hosted on my site, from their servers. Any javascript or html in this script will not execute since it isn’t a browser visiting. Some authentication stuff happens, and if my IPN php script determines everything is valid then I call PiwikTracker.php with the goal ID and revenue amount. At this point I have access to the custom variable I mentioned in step 1, and could pass it to PiwikTracker also.

If this can happen it would be a true solution, since paypal will keep hitting my IPN script until it see’s a proper response. Thus, 100% of the sales should get properly tracked.

I know enough php to be dangerous, but don’t consider myself a true developer. If however there is anything I can do to assist with getting this feature added (testing?) I would be more than happy.


(Matthieu Aubry) #4

thinking aloud:

For Piwik to track the conversion to the right visitor, and also attribute the conversion to the referer, we must call Piwik Tracking API

  • pass the visitor IP. to Piwik Tracker. The IP could be fetched by your website and passed to Paypal. Then it can be passed to Piwik via the PHP Tracking method

setIp()

http://qa.piwik.org:8080/phpdocs/PiwikTracker/PiwikTracker.html#setIp

Alternatively, we could also use the VisitorId as you suggest, which can be fetched in Javascript using


piwikTracker.getVisitorId()

but currently the Tracking API does not accept the visitor ID like it does accept the visitor IP.

  • pass to Tracking API the referer the visitor used to find the site in the first place (or the last referer used depending if this function is called in the JS API [quote=“Piwik Javascript tracking”]setConversionAttributionFirstReferrer( bool ) – Set to true to attribute a conversion to the first referrer. By default, conversion is attributed to the most recent referrer.[/quote] )

Currently, there is no JS helper function to fetch this referer value (stored in a first party cookie), but we could easily add it.
Also, there is no currently a way to pass this referer value to the Tracking API, but again we could add it easily as well.

So the missing pieces in Piwik are at least:

  • fetch the referer from cookie in Javascript (new JS function)
  • pass the referer to Piwik via tracking API (new Tracking API parameter & new function in PiwikTracker PHP class)

What is the current code snippet you use to trigger the conversion in the IPN script?


#5

I could pass the IP, and was looking at that. Would that be enough for piwik to associate the goal with the visitor? I suspect the visitorID would be more robust if we can implement that.

Goal triggering code snippet:


//Piwik Analytics
$piwikTracker = new PiwikTracker( $idSite = 1 );

// Sends Tracker request via http
$piwikTracker->doTrackPageView('Thank you!');

// You can also track Goal conversions
$piwikTracker->doTrackGoal($idGoal = 1, $revenue = $cartSub);


(Matthieu Aubry) #6

I have created a ticket at: Paypal tracking: Tracking API improvements & how to · Issue #2222 · matomo-org/matomo · GitHub

Please post your comments there. It will not be done in the next few days, but in the next few weeks we shall work on this. PLease post there any other feedback etc. Thanks!


(Matthieu Aubry) #7

I have implemented the missing functionnality in Piwik to allow accurate Paypal tracking “after the fact”. Please review my proposal in: Paypal tracking: Tracking API improvements & how to · Issue #2222 · matomo-org/matomo · GitHub and post your comments there if it working for you. Ideally, it would be great if we could have a simple how - to (FAQ or small doc page) explaining how to do Paypal tracking with Piwik !


#8

Ok, time to admit I’m a bit over my head here. I think I could probably handle the coding in my IPN file since it’s php, which I’m at least somewhat capable of working with. Javascript is a total mystery to me so I’m floundering right from the start here. Not that that’s stopped me from trying. Here’s where I am:

1 - Being totally new to piwik in general it took me quite some time (and a few false starts) to figure out where to get the latest beta files so I could try your improvements out. I think I got my site upgraded properly to 1.2.5rc8, but on my goals tab I get an error (Undefined index: allow_multiple in path/to/piwik/plugins/Goals/Controller.php on line 136). Did I defeat myself before I even got started? Should I post the full backtrace?
2 - Am I understanding correctly that I can just add “visitorId = piwikTracker.getVisitorId();” and “attributionInfo = JSON2.stringify(piwikTracker.getAttributionInfo());” to the standard javascript tracker that I already have at the bottom of my order page? From what I can tell nothing is getting populated into those variables. Here’s what I’ve done - I’m hoping to take my existing variable jsCustom (a string containing several values delimited by ampersands) and tack on visitorID and attributionInfo (with ampersands between them), then send it as part of the hidden input “custom” in the form “ppbuynow”:


<!-- Piwik tracker customized for this page only -->
<script type="text/javascript">
var pkBaseURL = (("https:" == document.location.protocol) ? "https://mysite.com/piwik/" : "http://mysite.com/piwik/");
document.write(unescape("%3Cscript src='" + pkBaseURL + "piwik.js' type='text/javascript'%3E%3C/script%3E"));
</script><script type="text/javascript">
try {
var piwikTracker = Piwik.getTracker(pkBaseURL + "piwik.php", 1);
piwikTracker.trackPageView();
piwikTracker.enableLinkTracking();
visitorId = piwikTracker.getVisitorId();
attributionInfo = JSON2.stringify(piwikTracker.getAttributionInfo());
jsCustom = jsCustom + "&" + visitorId + "&" + attributionInfo;
document.ppbuynow.custom.value = attributionInfo;
} catch( err ) {}
</script><noscript><p><img src="http://theeasyfrench.com/piwik/piwik.php?idsite=1" style="border:0" alt="" /></p></noscript>
<!-- End Piwik Tracking Code -->

The above works fine if I don’t try to add visitorID and attributionInfo to the jsCustom variable, so I don’t think I’ve done anything Too daft…?

FYI, if I can get this working I will be more than pleased to test it and draft a how to guide for paypal users. I appreciate your help on this.


#9

Update: since I only have a couple weeks of data in piwik and still have GA running for now, I reinstalled from scratch. So, disregard my first point above about the db troubles, that is resolved. Still don’t seem to be getting any data in the visitorID and attributionInfo variables though, not sure what I’m doing wrong…?


(Matthieu Aubry) #10

Don’t worry it is pretty complicated stuff and I didn’t explain properly :slight_smile:

The code you pasted above, is the first part of the problem: it is getting the values from Piwik, about the visit, before it is leaving your site to Paypal.

Now there must be a way, that Paypal will somehow contact your site later on (this is the IPN script I think?).

Then, when Paypal contacts you, you will want to execute a PHP Script that will track the Goal conversion for this visit.

To do so, the php script executed when paypal contacts you after the sale was done, will use the Tracking API and look something like:


require_once "/path/to/PiwikTracker.php";
PiwikTracker::$URL = 'http://www.example.org/piwik/';

$t = new PiwikTracker( $idSite = 1 );

// If you want to force the visitor IP, or force the server date time to a date in the past,
// it is required to authenticate the Tracking request by calling setTokenAuth with the Super User token_auth
$t->setTokenAuth( $token_auth );
$t->setVisitorId( $visitorId );
$t->setAttributionInfo( $attributionJsonEncoded );

// Track the goal with the revenue
$t->doTrackGoal( $idGoal = 1, $revenue);

As you see, to run this script you will need few information:

  • your super user token_auth, Piwik URL, Piwik idSite, and idGoal for your “Paypal order” goal
  • the $visitorId (which is the value that you put in your jsCustom variable - how can you get it back when this script runs?)
  • the $attributionInfo (same as $visitorId, you must get this value when Paypal contacts you back)
  • the $revenue that Paypal will also give you

I don’t know how you can get the data back from Paypal but hopefully you can find the information. If you are really lost I could do some research. Let us know how it goes!


#11

Actually that is the part I think I can handle. What’s throwing me is that first step - the javascript variables visitorId and attributionInfo appear to be empty (zero length empty strings). If I don’t try to add the visitorId and attributionInfo to my existing jsCustom variable, then I get it back at my IPN script as expected (string: “value1&value2&value3”) which I can then do a php explode on and do various things with those variables. If I try to append the visitorId and attributionInfo the whole jsCustom variable dissapears. If I try to do a document.write(visitorId) or document.write(attributionInfo) at the order page, nothing prints. If I pass visitorId to a test php script and do a var_dump I just see “NULL”. So, even though I’ve completely reinstalled my whole setup to be 1.2.5rc8, did I somehow miss something I needed so that these new javascript functions will work correctly?

I’ll send you some additional troubleshooting details in a PM.


(vipsoft) #12

It could be a length limit imposed by paypal.

Or maybe it’s an encoding problem?


document.ppbuynow.custom.value = encodeURIComponent(attributionInfo);


#13

I had some document.write statements to help debug, apparently they were creating problems. Thanks to some help from Matt in a PM thread, step 1 is now working properly for me. I’ll give step 2 a go this evening (expect it to work fine) and let everyone know.


#14

Ok, I’m misunderstanding something about the whole setTokenAuth thing. I had no trouble passing my variables to paypal and getting them back again, then seperating them out (had my script email them to me seperately to be sure, got exactly what I expected). However, the goal tracking is not working as expected. While trying to debug, I discovered that if my doTrackGoal statement was BEFORE my setTokenAuth, then at least a goal would be registered (although not properly attributed). However, if I moved my doTrackGoal statement to AFTER my setTokenAuth, then nothing worked. I’m pretty sure I have the right token (top of my page, clicked API). I even tried changing my password to force a new token, then tested with that and got the same results. I suspect once I can get properly authenticated as super-user then this puppy should work. Here’s my code:


// -- Piwik Tracking API init --
require_once "PiwikTracker.php";
PiwikTracker::$URL = 'http://mysite.com/piwik/';

echo("Testing.  Custom: ");

$temp = $_POST['custom'];

echo($temp);             // french or spanish & [affiliate code] & ip & visitorId & attributionInfo

list($custom, $affCode, $custIP, $visitorId, $attributionInfo) = explode("&", $temp);
list($affID, $prodID) = explode(":", $affCode);

echo("<br><br>visitorId: ");
echo($visitorId);
echo("<br><br>visitorId: attributionInfo");
echo($attributionInfo);

//Piwik Analytics
$piwikTracker = new PiwikTracker( $idSite = 1 );

$piwikTracker->setTokenAuth("my token from API section of piwik install");
//$piwikTracker->setIp($custIP);
//$piwikTracker->setCustomVariable (2, 'testIP', $custIP);
//$piwikTracker->setUrl($url='http://www.mysite.com/myPage.php');

$piwikTracker->setVisitorId($visitorId);
$piwikTracker->setAttributionInfo($attributionInfo);

$piwikTracker->doTrackGoal($idGoal = 1, $revenue = 25);

$piwikTracker->doTrackPageView('Display paypal post details');

Note that the above code is from a test page I put together to skip the hassle of going through paypal, since I’ve already verified that part will work fine.

Can anyone tell me what I’m doing wrong?


(Matthieu Aubry) #15

morrack, you are right that it should work as expected,

Would you be able to prepare a page, that would call the API with the setTokenAuth() and doTrackGoal to isolate the issue? Does this page also fail to track the goal?

Then my next step would be to enable debug as per instructions in Tracking API - Analytics Platform - Matomo

Then I would take a look at the piwik.php request output, and see what the debug messages say that piwik is doing.

I can help with this if you PM me the details and more info. You are very close it will work :wink:


#16

Hi,
seems like same problem. Even after upgrade to 1.3. still getting incorrect referer type (3 instead of 6) for campaign conversions and keyword column in database is empty. Any thoughts or ideas?

THX


(Matthieu Aubry) #17

vaso007, campaign tracking should now workin fine in 1.3, we resolved all issues pointed out in this thread (by PM).

What is the problem exactly?


#18

Hi,
I’m running five campaigns in same time but campaign goal count is zero although conversions are recorded in database. The problem is that Piwik identifies thous conversions as website goals (‘referer_type’ in piwik_log_conversion table is set to 3 instead 6 as i mentioned in post before) instead of campaign goals. As you can see on attached screenshot (attach_1.png), goal segmentation table shows strange results where campaign with zero visits shows conversion ratio higher than null which is obviously incorrect value. Meanwhile website conversions split table (attach_2.png) shows conversions of campaigns (referrer is of campaign site), again with strange visit count.

Thank you for advice.


(Matthieu Aubry) #19

Please ignore the column “Conversion rate” as it is not valid, and has been removed in SVN. See the other post: 301 Moved Permanently

Regarding 0 visits but conversions, this is possible, see the (new) FAQ: General - Analytics Platform - Matomo

let me know if you still have some questions


#20

Hi,
thank you for fast answer but it is still not solving my problem. I’m 100% sure that conversions from campaigns were there (I can see them in database table) but are not shown in segmentation report. Here is my “Conversions segmentation report” taken from UI (I’m omitting figures like conversion rate and so):
-------------------------- visits - conversions-
Search Engines 480 3
Direct Entry 160 6
Campaigns 110 0
Websites 39 7

And report should be:
-------------------------- visits - conversions-
Search Engines 480 3
Direct Entry 160 6
Campaigns 110 3
Websites 39 4

Again, some of conversions recorder as “Website conversions” should be recorded as “Campaign conversion”. In this case 3 conversions.