Search engines in the list of websites

Pete_Trombone · May 5, 2020, 2:46pm

Hello,

can anyone explain why the URL’s of the search engines are displayed under the channel types in the websites?
And why is the number of visits there significantly higher than the visits under the channel type search engines?
Doesn’t Matomo recognize every reference of a search in a search engine?

Regards
Pete

Lukas · May 5, 2020, 6:41pm

Hi,

Normally Matomo should detect the search engine if the referrer matches an entry in this list:

github.com

matomo-org/searchengine-and-social-list/blob/master/SearchEngines.yml

1.cz:
  - 
    urls:
      - 1.cz
    params:
      - '/s\/([^\/]+)/'
      - q
    backlink: 's/{k}'
    charsets:
      - iso-8859-2
118 700:
  - 
    urls:
      - www.118700.se
      - foretag.118700.se
      - webben.118700.se
    params:
      - q
    backlink: 'sok.aspx?q={k}'
123people:

This file has been truncated. show original

I can’t think of a reason why this would not work. Is this a new change? Maybe it is a bug.

Pete_Trombone · May 7, 2020, 11:10am

Hi Lukas,

when I look at the tracking data of the last years in Matomo, it always looks the same as above.
Although the individual search engines websites appear in SearchEngines.yml (e.g. www.bing.com, www.ecosia.org), they are listed as a website.
It is different with Google. Strangely enough I did not find the following Google URL’s in SearchEngines.yml:
www.google.com
www.google.de

I’m a little at a loss

Lukas · May 7, 2020, 12:09pm

Hi,

‘google.{}’ matches all google.tld subdomains.

urls

Each configuration needs to contain one ore more urls. Please only define the hostname. You can use {} as a placeholder for country shortcodes in subdomains or tld.

{}.searchengine.com would also match de.searchengine.com or nl.searchengine.com

searchengine.{} would also match searchengine.de or searchengine.nl

Notes:

For TLDs only {} would also match combined TLDs like co.uk . (Full list com.*, org.*, net.*, co.*, it.*, edu.* )

The first URL will be used for the icon, so the most popular/representative URL should be placed there

https://github.com/matomo-org/searchengine-and-social-list

Pete_Trombone · May 7, 2020, 12:40pm

Hm…
but I didn’t find an entry for Google like this:
www.google.{}
or
{}.google.{}

And why do the other search engines websites appear in the channel type “websites”?

Lukas · May 7, 2020, 12:44pm

Okay, I worded this a bit badly:
There is an entry here:

github.com

matomo-org/searchengine-and-social-list/blob/336fa41739715cc734e17f5c700ef7c7fd9fe40f/SearchEngines.yml#L875


    urls:
      - search.goo.ne.jp
      - ocnsearch.goo.ne.jp
    params:
      - MT
    backlink: 'web.jsp?MT={k}'
Google:
  - 
    urls:
      - google.com
      - 'google.{}'
      - encrypted.google.com
      - www2.google.com
      - ipv6.google.com
      - go.google.com
      - wwwgoogle.com
      - 'wwwgoogle.{}'
      - gogole.com
      - 'gogole.{}'
      - gppgle.com
      - 'gppgle.{}'

So Matomo should automatically group all referrers that look like google.something into the search engine category. If it doesn’t (as it looks like for you), then this is maybe a bug.

Pete_Trombone · May 7, 2020, 2:12pm

Okay, I understand.

Then it’s a bug. And now?

Lukas · May 7, 2020, 2:17pm

Could you please create a bug report with as many details as possible:

Pete_Trombone · May 7, 2020, 2:53pm

Done

github.com/matomo-org/matomo

Search engines are displayed in the wrong channel type

opened 02:52PM - 07 May 20 UTC

closed 08:09PM - 14 May 20 UTC

PeteTrombone

Bug not-in-changelog

Search engine websites are displayed under the channel type "Websites". ![ima…ge](https://user-images.githubusercontent.com/57910876/81308684-5a19a600-9082-11ea-9546-43fd22bf9002.png) In addition, there are also entries of the search engines under the channel type "Search Engines". However, these are completely different values. ![image](https://user-images.githubusercontent.com/57910876/81308923-a6fd7c80-9082-11ea-8c00-42c667d4d1d1.png) When I look at the tracking data of the last years in Matomo, it always looks the same as above.

progger · November 18, 2020, 12:21pm

Hello,

I recognized the same issue with the latest Matomo v3. www.startpage.com and eu.startpage.com are classified as websites and not as a search engine. This happens even General['enable_internet_features'] is not disabled here.

So I tried the console command mentioned in the Github issue:

console core:run-scheduled-tasks --force "Piwik\Plugins\Referrers\Tasks.updateSearchEngines"

The task is successfully executed but this does not solve the problem. As an alternative an SQL query was mentioned to delete the SearchEngineDefinitions field.

But I’d like to ask first if this could be really the solution. I don’t want to mess up the Matomo installation completely with executing raw queries on the database. It would feel more comfortable solving this with the console command so I wonder why this does not seem to have any effect?

Lukas · November 18, 2020, 1:53pm

Hi @progger,

This is caused by the following:

Matomo uses this list for detecting search engines:

github.com

matomo-org/searchengine-and-social-list/blob/9d2cfff43c899f2aed0f94df7496537f830b3940/SearchEngines.yml#L2009-L2036

      
        
            StartPage:
              -
                urls:
                  - startpage.com
                  - classic.startpage.com
                  - www.startpage.com
                  - eu.startpage.com
                  - s6-eu5.ixquick.com
                  - s2-eu4.startpage.com
                  - s6-eu4.startpage.com
                  - s7-eu4.startpage.com
                  - s1-eu5.startpage.com
                  - s2-eu5.startpage.com
                  - s4-eu5.startpage.com
                  - s5-eu5.startpage.com
                  - s6-eu5.startpage.com
                  - s7-eu5.startpage.com
                  - s8-eu5.startpage.com
                  - s10-eu5.startpage.com
                  - s11-eu5.startpage.com

This file has been truncated. show original

But as specified it only detects startpage if either the referrer contains a ?query=searchterm part of if the referrer is /do/asearch. As startpage does neither anymore, but uses a referrer policy to tell the browser to set the referrer to https://www.startpage.com/ Matomo doesn’t detect it.

I wrote down my thoughts about this in a new issue:

github.com/matomo-org/matomo

Use `hiddenkeyword` for Search engines by default

opened 01:57PM - 18 Nov 20 UTC

closed 02:49AM - 31 Dec 20 UTC

Findus23

c: Usability RFC

while this mostly affects https://github.com/matomo-org/searchengine-and-social-…list/ I think this should be changed in Matomo itself reported in https://forum.matomo.org/t/search-engines-in-the-list-of-websites/37192/12?u=lukas Matomo doesn't detect a lot of search engines anymore as the detection assumes that the referrer URL contains the search query specified in the search engine list. But hardly any search engine sends it anymore, so it doesn't work. As a workaround it seems like the `hiddenkeyword` keyword was added to the yaml file to specify the URL that is used to do the redirect over and which will be then the referrer URL. But as referrer policy and such things are now widespread, no one opens links via redirects anymore and this is again pointless. One could fix this issue by adding ```yaml hiddenkeyword: - '/^$/' - '/' ``` to every search engine in the list, but at this point the setting becomes useless and instead I think Matomo should instead handle every referrer URL like `https://www.startpage.com/` as a search engine independent on the exact URL or query parameters.

progger · November 18, 2020, 4:09pm

Hi @Lukas,

that’s an interesting issue, I just saw startpage.com was in the YML file so it seemed related to this thread but this is indeed a separate problem.

Thanks for analysing it and creating the Github issue. I also thing search engines should just be identified by the domain part in future. This would be also easier to maintain.