Problems with IIS and Piwik

Support & Bugs
1

Hello people.

Im using Piwik in Linux and Windows and I have a lot of problems running it on IIS.

The problems are always of the same style:

2013-05-08 12:51:16,450: [DEBUG] Error when connecting to Piwik:
1000 lines parsed, 200 lines recorded, 8 records/sec (avg), 0 records/sec (curre
nt)
1000 lines parsed, 200 lines recorded, 8 records/sec (avg), 0 records/sec (curre
nt)
2013-05-08 12:51:18,490: [DEBUG] Error when connecting to Piwik:
Fatal error: None
You can restart the import of “c:\u_ex130503.log” from the point it failed by sp
ecifying --skip=200 on the command line.

I have tried with IIS 7.5 and 8. PHP 5.2, 5.3 and 5.4, little logs (rarely crashes) and medium or large logs (frecuently crashes), python 2.6 and python 2.7…always this problem. I include download and install piwik with IIS Wep Platform…the same happens.

Another thing is that if I process the logs in the linux system, redirecting them to the Windows system, the process fails too, but, if I process THE SAME LOG in the windows system to the Linux System (with Piwik installed) or ALL THE PROCESS in the Linux System, ALL GOES WELL and the NEVER HAPPENS.

I think that its an IIS+ PHP configuration, but I dont get better results. I attach the log to verify it, but in Linux System it is 100% processed without the error.

Any ideas?

Thanks a lot and sorry by my English.

PS:
The file is a RAR with .pdf added to avoid the extension restrictions.

2

Had the same problem and I assume this is a bug. Using the recommended releases: Piwik:1.11.1 Python:2.7.4 PHP:5.3.25

Cause of Fatal Error: Piwik returns a string that is not expected to the log_parser.py

Should be the 1x1 GIF file
PIWIK_EXPECTED_IMAGE

But Piwik replies with
{“succeeded”:0,“error”:null}

The succeeded value may change.

IIS is OK, SQL is OK, etc.
So I chose a brute force method to ignore errors only for the bulk load calls:

In import_logs.py:
Add the 2 lines of code below: #Brute force ignore errors
to assume all requests are good.

Crude but effective. Unsure if this loses any data.

@staticmethod
def _call_wrapper(func, expected_response, on_failure, *args, **kwargs):
    """
    Try to make requests to Piwik at most PIWIK_FAILURE_MAX_RETRY times.
    """
    errors = 0
    while True:
        try:
            response = func(*args, **kwargs)

# Brute force ignore errors
if expected_response == PIWIK_EXPECTED_IMAGE:
	response=PIWIK_EXPECTED_IMAGE

            if expected_response is not None and response != expected_response:
3

Can you try with latest beta: Index of / wheret his bug should be fixed?

4

Even better… the latest build 1.12b14 fixed this bug!

5

I tried it with the latest build and nothing changes…

C:>python d:\piwik\misc\log-analytics\import_logs.py --url=http://localhost:888
8 d:\piwik\Log\W3SVC4\u_ex130503.log --idsite 1 --enable-http-errors --enable-ht
tp-redirects --enable-static -ddd
2013-05-13 09:06:34,236: [DEBUG] Accepted hostnames: all
2013-05-13 09:06:34,236: [DEBUG] Piwik URL is: http://localhost:8888
2013-05-13 09:06:34,236: [DEBUG] No token-auth specified
2013-05-13 09:06:34,237: [DEBUG] No credentials specified, reading them from "d:
\piwik\config\config.ini.php"
2013-05-13 09:06:34,240: [DEBUG] Using credentials: (login = admin, password = 8
34d3d5fa57472213b8d3be0dfac869f)
2013-05-13 09:06:34,382: [DEBUG] Authentication token token_auth is: 0e1e98beb69
e8cd0ab7db8df06404a2e
2013-05-13 09:06:34,440: [DEBUG] Resolver: static
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2013-05-13 09:06:34,556: [DEBUG] Launched recorder
Parsing log d:\piwik\Log\W3SVC4\u_ex130503.log…
2013-05-13 09:06:34,558: [DEBUG] Detecting the log format
2013-05-13 09:06:34,562: [DEBUG] Format iis matches
2013-05-13 09:06:34,562: [DEBUG] Format s3 does not match
2013-05-13 09:06:34,562: [DEBUG] Format common_complete does not match
2013-05-13 09:06:34,563: [DEBUG] Format common does not match
2013-05-13 09:06:34,563: [DEBUG] Format common_vhost does not match
2013-05-13 09:06:34,565: [DEBUG] Format ncsa_extended does not match
2013-05-13 09:06:34,565: [DEBUG] Format iis is the best match
2013-05-13 09:06:34,568: [DEBUG] Invalid line detected (line did not match): #So
ftware: Microsoft Internet Information Services 7.0

2013-05-13 09:06:34,571: [DEBUG] Invalid line detected (line did not match): #Ve
rsion: 1.0

2013-05-13 09:06:34,572: [DEBUG] Invalid line detected (line did not match): #Da
te: 2013-05-03 00:04:09

2013-05-13 09:06:34,575: [DEBUG] Invalid line detected (line did not match): #Fi
elds: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-quer
y s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-ho
st sc-status sc-win32-status sc-bytes cs-bytes time-taken

824 lines parsed, 200 lines recorded, 199 records/sec (avg), 200 records/sec (cu
rrent)
2013-05-13 09:06:35,953: [DEBUG] Error when connecting to Piwik:
1004 lines parsed, 200 lines recorded, 99 records/sec (avg), 0 records/sec (curr
ent)
1004 lines parsed, 200 lines recorded, 66 records/sec (avg), 0 records/sec (curr
ent)
2013-05-13 09:06:37,989: [DEBUG] Error when connecting to Piwik:
1004 lines parsed, 200 lines recorded, 49 records/sec (avg), 0 records/sec (curr
ent)
1004 lines parsed, 200 lines recorded, 39 records/sec (avg), 0 records/sec (curr
ent)
2013-05-13 09:06:40,022: [DEBUG] Error when connecting to Piwik:
Fatal error: None
You can restart the import of “d:\piwik\Log\W3SVC4\u_ex130503.log” from the poin
t it failed by specifying --skip=204 on the command line.

3 error and then exits with error…

I tried changing the .py to update the retries from 3 to 30, but the problem is the same…starts to fail and fails fails fails until exit with the same error.

I tried changing the application_pool on IIS becouse I think that is something of the PHP on IIS. I tried to redirect the query to another Linux piwik instance (in other machine) and it works…executing it on the same Windows machine (see the next log). I think that its something of the Py-PHP working on IIS.

The same LOG in the same machine…but redirecting the calls to another Piwik machine (working in Linux):

C:>python d:\piwik\misc\log-analytics\import_logs.py --url=http://XXXXXXXXXXXXXX:8888 d:\piwik\Log\W3SVC4\u_ex130503.log --idsite 1 --enable-http-errors --enable-http-redirects --enable-static -ddd

…some lines removed to make it more readable…

5821 lines parsed, 5200 lines recorded, 156 records/sec (avg), 200 records/sec (
current)
6005 lines parsed, 5200 lines recorded, 152 records/sec (avg), 0 records/sec (cu
rrent)
2013-05-13 09:10:42,805: [DEBUG] Invalid line detected (line did not match): 201
3-05-03 09:46:56 W3SVC4 entorno125 ::1 GET /themes/default/images/success_medium
.png - 8888 - ::1 HTTP/1.1 Mozilla/5.0+(compatible;+MSIE+9.0;+Windows+NT+6.0;+WO
W64;+Trident/5.0) PIWIK_SESSID=tnk8vh9brlfrn9vb0a7m74m0l3 http://localhost:8888/
index.php?action=tablesCreation&module=Installation localhost:8888 304 0 210 522
0

2013-05-13 09:10:42,809: [DEBUG] Invalid line detected (line did not match): 201
3-05-03 09:47:16 W3SVC4 entorno125 ::1 GET /index.php action=generalSetup&module
=Installation 8888 - ::1 HTTP/1.1 Mozilla/5.0+(compatible;+MSIE+9.0;+Windows+NT+
6.0;+WOW64;+Trident/5.0) PIWIK_SESSID=tnk8vh9brlfrn9vb0a7m74m0l3 http://localhos
t:8888/index.php?action=tablesCreation&module=Installation localhost:8888 200 0
7940 433 983

6112 lines parsed, 5400 lines recorded, 153 records/sec (avg), 200 records/sec (
current)
6112 lines parsed, 5600 lines recorded, 154 records/sec (avg), 200 records/sec (
current)
6112 lines parsed, 5800 lines recorded, 156 records/sec (avg), 200 records/sec (
current)
6112 lines parsed, 6062 lines recorded, 158 records/sec (avg), 262 records/sec (
current)
Purging Piwik archives for dates: 2013-05-03
To re-process these reports with your new update data, execute the piwik/misc/cr
on/archive.php script, or see: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo for more i
nfo.

Logs import summary

6062 requests imported successfully
1812 requests were downloads
50 requests ignored:
    50 invalid log lines
    0 requests done by bots, search engines, ...
    0 HTTP errors
    0 HTTP redirects
    0 requests to static resources (css, js, ...)
    0 requests did not match any known site
    0 requests did not match any requested hostname

Website import summary

6062 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 38 seconds
Requests imported per second: 158.62 requests per second

Works fine…Why on Windows dont do it??? :(((((((

6

Can you summarize the problem clearly, I don’t understand ?

7

Hi matt,

Sorry, I tried to give more info to make the problem more abordable, but I see that I make it more complicated.

The problem is the same that I said in the first post:

When I process the logs in a Windows System with IIS+Python+PHP…the process Crashes. Three and ends with Fatal error: None

The las post…was only to give more info becouse the same log processed in Linux with Apache+Python+PHP goes well…and the same Log processed in Windows, but redirecting the process calls to a Linux system goes well too. The same log, processed in Linux,but redirecting the info to a Windows System crashes with the same error (Three and ends with Fatal error: None ). This is why I think that it is a PHP problem.

All the requeriments in Windows systems are OK, include I use a preconfigured Piwik installed with IIS Web Platform Installer and exactly the same happens.
In my first post I attach an example Log to see the problem. In the second post you can see the parameters I use to call the .py script (I used other structures to try, but with all the parameters get the same error).

I hope that the problem were more clearly now.

8

I have tried to tuning the fastcgi and other PHP settings on IIS but nothing changes…

I can`t go ahed with this…any help will be wellcome.

Thanks!

9

when you get error, check the webserver error log files, where the requests are sent to. Maybe it will contain some indication of why the request has failed.

10

It dont give so much info:

#Date: 2013-05-16 06:28:40
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-win32-status sc-bytes cs-bytes time-taken
2013-05-16 06:28:40 W3SVC4 XXXXXXX 127.0.0.1 POST / - 8888 - 127.0.0.1 HTTP/1.1 Python-urllib/2.7 - - localhost:8888 200 0 470 304 2839
2013-05-16 06:28:40 W3SVC4 XXXXXXX 127.0.0.1 POST / - 8888 - 127.0.0.1 HTTP/1.1 Python-urllib/2.7 - - localhost:8888 200 0 782 297 182
2013-05-16 06:28:41 W3SVC4 XXXXXXX 127.0.0.1 POST /piwik.php - 8888 - 127.0.0.1 HTTP/1.1 Python-urllib/2.7 - - localhost:8888 200 0 311 67280 801
2013-05-16 06:28:41 W3SVC4 XXXXXXX 127.0.0.1 POST /piwik.php - 8888 - 127.0.0.1 HTTP/1.1 Python-urllib/2.7 - - localhost:8888 200 0 225 86890 342
2013-05-16 06:28:44 W3SVC4 XXXXXXX 127.0.0.1 POST /piwik.php - 8888 - 127.0.0.1 HTTP/1.1 Python-urllib/2.7 - - localhost:8888 200 0 224 54030 19
2013-05-16 06:28:46 W3SVC4 XXXXXXX 127.0.0.1 POST /piwik.php - 8888 - 127.0.0.1 HTTP/1.1 Python-urllib/2.7 - - localhost:8888 200 0 224 54030 22

All goes well (at log level)…there isn`t any php error log…im testing it in the same local machine…