Referrer problem after update

Support & Bugs
1

hello

after updated piwik to new version google spamming in my Referrer Websites list

see screenshot please

image
image697×3466 256 KB

how i can ignore google as referrer website

2

is there anyway to go back to previous version?
this is very annoying

3

Hello. I have the same problem after updating from 2.16.1 to 2.16.2. See attached screenshot. The newest version don’t recognize Google search engine and treat it as normal website.

I update Piwik by manually downloading new version archive and replacing all files. I don’t use Piwik update feature.

What I tried, tested and this not fixed problem:

  1. Manually updated file [piwik]/vendor/piwik/searchengine-and-social-list/SearchEngines.yml from https://github.com/piwik/searchengine-and-social-list and ensured that this file has write access by server.
  2. Emptied [piwik]/tmp folder.

4

Seems something broke the referrer detection. I’ll try to have a closer look at that later.

5

There was an indention mistake in the search engines definition file. I’ve just fix that. Could anyone check if that solves the problem?

6

I installed locally fresh copy and it works. I also copied whole database from server to this installation and still works - without your last fix.

I think, problem is related to some nonstandard config, maybe tracker. I’m trying to reproduce it and I’ll let you know when find reason.

7

no
problem still exits

do we have to replace a file or something?

8

Within the release, there is another version of SearchEngines.yml. Piwik automatically pulls a new version from the repos master from time to time. Maybe your Piwik installation updated the file when it was “broken”.

9

OK, I got it. It seems that database didn’t update correctly after upgrade to 2.16.2, or even on earlier versions.

I was trying to find a difference between clean install (working) and my install (broken). It was not easy, but finally found a difference in database field piwik_option.SearchEngineDefinitions. It’s a serialized and base64 encoded array with search engines.

In my database (updated to 2.16.2) is outdated version of this data and in clean install is a new version. The main difference is that new version has “hiddenkeyword” array on Google search engines definition.

On version 2.16.2 there was a change how Piwik deal with empty keywords - in most cases Google passes only main domain (https://www.google.com/) as referrer. If you see here Improve detection for empty keywords for referring search engines by sgiehl · Pull Request #10163 · matomo-org/piwik · GitHub you will see changes in plugins/Referrers/SearchEngine.php file.

That’s why after update, old SearchEngineDefinitions stopped working. To confirm this, I changed SearchEngineDefinitions to old version in clean install and problem appeared.

After manually changing SearchEngineDefinitions I also emptying [piwik]/tmp folder.

I looked into code and it works as follows:

  • Piwik gets search engine definitions from database field SearchEngineDefinitions.
  • If SearchEngineDefinitions don’t exists, read from file SearchEngines.yml and stores it in database. So if SearchEngineDefinitions exists, file SearchEngines.yml is not used.
  • Once a week search engines should be updated automatically. File SearchEngines.yml is downloaded from internet and stored in database as SearchEngineDefinitions, local file is not updating.

So the problem is not with Piwik upgrade but with automated updating SearchEngines.yml. My data was very old, there are no “hiddenkeyword” array in it. For some reason this task don’t run.

Here is a quick fix to repair broken referrers, for others who have the same problem after update to 2.16.2.
But this don’t solve the source of problem, I don’t know, why search engine definitions are not updated automatically.

  • First ensure you have updated Piwik files to new version.
  • On table piwik_option find record where option_name=SearchEngineDefinitions and delete it. Remember, BEFORE any changes in database MAKE A BACKUP, at least piwik_option table.
  • Clear Piwik temporary files, [piwik]/tmp folder.
  • When someone enter your website with referrer, Piwik will create SearchEngineDefinitions record based on your local file SearchEngines.yml.