Report search engine visits?


#1

Hi Friends,
Even though I have Piwik installed and working, there are still some things that are just over my head. So this may well be a newbie question. But I have searched the faq and forum, and didn’t find an answer.

Anyway, it seems like the Search Engines widget is for reporting when a visitor’s referring site is a search engine. Or in other words, the visitor got to my site via search engine. But the visitor is a person.

So far, I haven’t figured out how to know when a search engine’s spider/crawler/whatever is visiting my site. It seems like there should be a way to detect those, because my Simple Machines Forum has a mod for that. So I do get some very basic info about those visits – the date they visit and how many pages they visit each time… But I’ve been thinking it would be interesting to see how they navigate the site and to have some statistics about the search engine visits, such as how many per day, week, month, etc., average page hits per visit, or average hits per spider per visit, etc.

Is there already a way to track search engine spiders? If so, how? And if not, is it worth requesting as a new feature? I don’t know, maybe most Piwik users are so advanced in their knowledge and skills that search engine spiders aren’t worth mentioning. But it seems like if could be interesting to watch.

Thanks for your comments :slight_smile:


#2

Hi brynn,

I have same question wtih you :slight_smile: Today I searched but i did not find any information about this topic.

I want to ask how i can learn When search engine’s spider/crawler etc coming visiting my website.

Another Question ,

If i refer the google search engine’s spider/crawler for come to visit my website. When google search engine’s spider/crawler will come ?


(Matthieu Aubry) #3

Maybe this plugin could help you? Plugin: BotTracker to track bot-actions · Issue #2391 · matomo-org/matomo · GitHub

otherwise to see the list of bots, you can also import the server logs and set the option --enable-bots


#4

Oh awesome! Thanks matt :slight_smile:

Couple questions. Are there other plugins which don’t come packaged with Piwik, where we can search, instead of posting a topic about it?

Oh wait, I just found Piwik Marketplace (http://plugins.piwik.org/). I guess that’s where the other plugins are? Otoh, it doesn’t list this BotTracker plugin there, so maybe not. :S

Also, I can’t figure out which file to download. I see a few files under “Attachments” but if that’s where the file is, I don’t know which one I need. Or, I also notice Register and Login links at the very top of the page. Maybe I need to be registered and logged in to see the download link?

Thanks again.


umutsuvarisi, I’m not sure what you mean about referring google’s spider to come and visit your site. But in my experience, my site was only online for a couple of days, before google found it. I have no idea how the google spider knew it was there, because I had not announced the site anywhere yet, at that time. Anyway, from my forum’s search engine mod, it looks like google visits every day! It doesn’t look at every page every day, and I don’t know how it decides which pages to look at. But it’s all very interesting, which is why I wanted to know if Piwik could provide more data.


(Matthieu Aubry) #5

BotTracker should be on: http://plugins.piwik.org/ but it isn’t yet. Stay tuned…


#6

(tu)


#7

Hi matt. Firstly thank you so much.

I want to ask another question. How can i learn google search engine’s spider/crawler etc visiting a web site… Not my website. I want to learn any website .


(Matthieu Aubry) #8

Check out this plugin: BotTracker Plugin: BotTracker to track bot-actions · Issue #2391 · matomo-org/matomo · GitHub

Download latest version from: GitHub - Thomas--F/BotTracker: BotTracker-Plugin for Piwik


#9

Oh great! Thanks matt :slight_smile:

One more question. At the bottom of the github page, it says

This plugin is currently in BETA-status, so I don’t recomment to use it in production until it’s well tested.

Does “use in production” mean use on a very busy site? And does “until it’s well tested” mean tested by me, on my site? Or is that an old message that no longer applies? Because I see that work was started on it 3 years ago. Maybe it’s no longer in beta?

Thanks again :slight_smile:

Edit – Another question. The file I found to download is BotTracker-master. But the instructions said the file is “BotTracker”. I wonder if I downloaded the right file?

I apologize for the confusion. I’m not familiar with GitHub. And this is for my first website. So I’ve had a lot to learn, lol. Do I have to register and login, to be able to download the correct file?

TA


(Matthieu Aubry) #10

On github, click DOWNLOAD ZIP.

In your Piwik, click “Marketplace” then “upload a plugin ZIP” and upload this ZIP.

Note: i havent tested


#11

Ok then, that’s exactly what I downloaded. I just wanted to be sure.

Thanks again.

Edit
PS – I’ll report my experienced after it’s installed, in case anyone else is curious :slight_smile:


#12

test

I’ve posted 2 further replies to this topic, 2 days ago. (really only 1, but I tried to post it a 2nd time too) I assume you’re starting to moderate for spam, because after I posted, there was a message about being a moderated forum. But 2 days doesn’t seem especially timely. I wonder if moderating on only new topics would be more effective? I don’t see effective spammers burying their spam in otherwise appropriate topics…although I know they do that sometimes.

Anyway, just my opinion. No offense intended :slight_smile:

Edit
Yikes – it posted!! What happened to my 2 previous replies???

Ok, I’ll try an post again, cross fingers :slight_smile:

Edit #2
Now I am NOT able to post another message. What the heck is going on? (I need more assistance, before I can give then promised review)

Edit #3
Ok, since I can edit, here comes the message I’ve been trying to post:

It seems like I have BotTracker at least partially installed, because it now appears in my plugins list, and I’ve clicked to Activate.

The next step in the instructions (here) says:
[ul]
[li]Configure the Bot-List by editing the MySQL-Table piwik_bot_db
[/li][/ul]
But I can’t find such a table. I’m looking in phpMyAdmin. Is there somewhere else I can look? Or am I supposed to create such a table, or maybe even new MySQL database? Maybe it will be fairly easy to figure out how to edit the list, once I find it. But I just don’t see it.

Also, there’s nothing new in my Piwik, except for the plugin being listed. I don’t see any new widgets or stats or info about search engine bots/spiders, at all. I know I haven’t set up the db properly, but the ticket seems to imply there are preconfigured bots (“Some well-known bots are pre-configured”) and I don’t see anything. And as I’ve previously mentioned, I know google visits every day.

Or should I be posting this as a Comment on the ticket?

Thanks for your help :slight_smile:

cross fingers, quick prayer, submit


#13

Ok, so I’ve apparently installed BotTracker-master successfully, or at least partially, because BotTracker now appears in my list of plugins, and I’ve clicked to Activate.

However, I can’t find any table called “piwik_bot_db” in my piwik db, which are part of the instructions here. But even if I could find it, I don’t know what it means to “Configure the Bot-List by editing the MySQL-Table piwik_bot_db”. Maybe it will be easy enough to figure out, if I could find it. I looked for it using phpMyAdmin, which is part of my server. Is there another place where it might be? Or did I do something wrong installing it? Since it shows up in my list of plugins, it seems like it was installed. Or am I supposed to create a table called piwik_bot_db, or maybe a whole new db?

Also, I don’t see any new widgets, or stats, or mention of search engine bots/spiders in my Piwik installation. So even though it’s at least partially installed, it doesn’t seem to be working.

Or should I be asking these questions on the BotTracker ticket (Plugin: BotTracker to track bot-actions · Issue #2391 · matomo-org/matomo · GitHub) now?

Cross fingers this get posted (3rd time’s a charm?)

Thanks again :slight_smile:


#14

Can anyone help? Or should I should post my difficulty installing BotTracker plugin, as Comments, to the ticket? (Plugin: BotTracker to track bot-actions · Issue #2391 · matomo-org/matomo · GitHub)


#15

2nd try

Can anyone answer this, or should I be posting in Comments area on the ticket?

Thanks.


(Matthieu Aubry) #16

If you have questions about specific plugins please ask in the plugin ticket or github etc. as the developers do not read this forum


#17

Ok, thanks :slight_smile:


#18

Sorry to resurrect this old topic. I happened to see this just now:

Using the Javascript Tracker, you can specifically tell Piwik to track the few advanced bots that will execute Javascript with _paq.push([‘appendToTrackingUrl’, ‘bots=1’]);

on this page: Does Piwik track visitors without Javascript? Does Piwik track search engine bots or other spider / robots? - Analytics Platform - Matomo. It must be new, since this topic was posted?

So I’ve added that to my js code. But now, I’m curious where to look to find a report.

Where will the results be found, when these advanced bots visit? Will they just be in the same visitor log, with all the “human visitors”?

Thanks for your help :slight_smile:


#19

High quality back links are still difficult to find for achieving top results in search engines. When we do that by keeping the continuously changing search engine algorithms and SEO strategies in mind, managing a successful link building campaign can be a challenging task.

best seo services