Logfile analyse - same log file but different visitor counts between Matomo and AWStats

lekis · September 22, 2022, 6:45am

Hello,
we have a Matomo installation for logfile analyse. AWStats and Matomo use the same logfile. The logfile importer script runs with the default option without import any additional data.
“python3 /path/to/matomo/misc/log-analytics/import_logs.py --url=https://stats.domain.com --idsite=1 --recorders=12”

But the counts of visitors show a big different between AWStats and Matomo. Matomo counts much more visitors as AWStats. For example August:
AWStats - 77.490
Matomo - 291.317

Anybody an Idea why?
Thanks in advance.

heurteph-ei · September 22, 2022, 7:50am

Do you have lots of visitor coming for example from the same company? (then the IP could be the same for several visitors, and Matomo could not distinct them…?)

lekis · September 22, 2022, 10:40am

Not really.
But in the log file are the IPs anonymized saved. However AWStats get the same datas from the log file to analyse.

heurteph-ei · September 22, 2022, 11:40am

Then I assume Matomo and AWStats don’t use the same algorithm to consider one single visitor. Is there any documentation about AWStats behavior?

lekis · September 22, 2022, 1:08pm

Thank you for your suggestion. I have found only this here:

And in this:
“FAQ-COM250 : DIFFERENT RESULTS THAN OTHER ANALYZER
PROBLEM:
I also use Webalizer, Analog (or another log analyzer) and it doesn’t report the same results than AWStats. Why ?
SOLUTION:
If you compare AWStats results with an other log file analyzer, you will found some differences, sometimes very important.
In fact, all analyzer (even AWStats) make “over reporting” because of the problem of proxy-servers and robots. However
AWStats is one of the most accurate and its “over reporting” is very low where all other analyzers, even the most famous,
have a VERY HIGH error rate (10% to 200% more than reality !)”

Is this really? Matomo shows actually more than 200% more … It is very heavy to believe but it looks to be true. Or?

heurteph-ei · September 22, 2022, 2:16pm

Can you check if you have headless browsers in your Matomo?

lekis · September 22, 2022, 2:34pm

Could you please help me where and how?
Or do you mean this?
matomo

heurteph-ei · September 22, 2022, 2:40pm

Or things like:

lekis · September 22, 2022, 2:59pm

Thank you.
We have only one daily “headless chrome”.

lekis · October 8, 2022, 6:28am

Is there no solution?
Is this statement true for Matomo??:
“Some log analyzers use the “Hits” to count visitors …”

Lukas · October 10, 2022, 9:50am

Hi,

Apart from the grouping from visits to visitors, there is also one more source of differences:

Matomo excludes everything it detects as bots. First very roughly in the import script (to keep the amount of data sent to Matomo low):

github.com

matomo-org/matomo-log-analytics/blob/fa173370f84052f100d88e7ed27f85750ea20208/import_logs.py#L84

      
        
                'ibooks jar json mpg mp2 mp3 mp4 mpeg mov movie msi msp odb odf odg odp '
                'ods odt ogg ogv pdf phps ppt pptx qt qtm ra ram rar rpm rtf sea sit tar tbz '
                'bz2 tbz tgz torrent txt wav webm wma wmv wpd xls xlsx xml xsd z zip '
                'azw3 epub mobi apk '
                'md5 sig'
            ).split())
            
            
# If you want to add more bots, take a look at the Matomo Device Detector botlist:
            # https://github.com/matomo-org/device-detector/blob/master/regexes/bots.yml
            # user agents must be lowercase
            EXCLUDED_USER_AGENTS = (
                'adsbot-google',
                'ask jeeves',
                'baidubot',
                'bot-',
                'bot/',
                'ccooter/',
                'crawl',
                'curl',
                'echoping',
                'exabot',

And then more precisely server-side using DeviceDetector.

lekis · November 24, 2022, 8:15pm

Hi Lukas,

thank you for your reply.
But I don’t think that the problem are the different bots / user agents.
I rather think the problem should be somewhere here (clicks or rewrites?):

If you compare AWStats results with an other log file analyzer, you will found some differences, sometimes very important. In fact, all analyzer (even AWStats) make “over reporting” because of the problem of proxy-servers and robots. However AWStats is one of the most accurate and its “over reporting” is very low where all other analyzers, even the most famous, have a VERY HIGH error rate (10% to 200% more than reality !).

Yes, we have a difference of 300%.

Some log analyzers use the “Hits” to count visitors. This is a very bad way of working : Some visitors use a lot of proxy servers to surf (ie: AOL users), this means it’s possible that several hosts (with several IP addresses) are used to reach your site for only one visitor (ie: one proxy server download the page and 2 other servers download all images). Because of this, if stats of unique visitors are made on “hits”, 3 users are reported but it’s wrong. So AWStats considers only HTML “Pages” to count unique visitors. This decrease the error, not completely, because it’s always possible that a proxy server download one HTML frame and another one download another frame, but this make the over-reporting of unique visitors less important.

or /and

AWStats does not count twice (with default setup) redirects made by server “rewrite rules”. Such rule makes two hits into log files, so most log analyzer count them twice, but only one page were “viewed”.
Then, there is internal bugs in log analyzers that make reports wrong.

Could one of the two statements also apply to matomo?

If not, I have to keep researching.
But if so, then I could understand the source of the difference.