So, a (pain-in-the-ass) client of mine wants his analytics in a very specific format.
Page Title | Hits | Unique Visitors | Country #1 Hits | Country #1 Unique Visitors | Country #2 Hits...
- Subpage Title | Hits | Unique | etc etc etc
, etc. It fetches statistics for the top three countries, the top three referrers, as well as raw hit count.
I have been able to set up a (probably horribly inefficient) system to pull all these statistics together, using the Page Titles report as a base and PHP foreach loops making custom segment requests for each set of specific stats.
It all works fine (if slowly) but at a certain point in the generation of the report, statistics stop coming in. At around the 11,000th data value or so, all statistics are filled in as blank, continuing until the end of the report.
Is there some limit that I’m bumping into that I’m unaware of?
They don’t archive, but I can start if that will allow me to generate the report better/more easily.
I can reproduce it fairly regularly. Looping through every page title and getting the country/referrer segments for each takes some hours, and the script updates the saved output after every row. At a certain point (again, around 11,000 or so cells in) the script just starts outputting blank cells. I’m not sure why it would work for several hundred rows of loops and then stop, but that’s what I’m trying to track down.
So, if I start archiving, would this method work better? Would there be another method available to me that would?
Piwik automatically archives data when you request it, but that is slow and could be a responsible for what you get (although Piwik should never return empty data).
Please let us know if this continues to be a problem.
I just setup auto-archiving, and I’ll let you know if it resolves the issue after the first archive.
If it doesn’t, would you be able to take a look at my PHP script? I know the code isn’t great (though it is commented as well as I could) and I’d love to get some tips from someone who actually knows the infrastructure.
I’ve been getting XML back from each API request and feeding them into SimpleXMLElement to be able to use them in the foreach loops. It dawns on me that maybe requesting it in a different format can save some effort, but it doesn’t seem like the original php object format is hierarchical like XML (and JSON, I presume) is. I see that there is an idsubdatatable_in_db set, which I assume references the later hierarchy, but I don’t know the API call to retrieve it. Adding &idSubtable={the idsubdatatable} as a parameter doesn’t seem to produce anything usable.
Any insight on whether I should keep at using XML (using isset($report->subtable) to drill through the hierarchy) or is there a more efficient option?