Issues with IIS log Importing


#1

Hi,

In addition to my other thread it seems I also have found issues with importing IIS log files. The following error is output when attempting to do this:

Traceback (most recent call last):
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1528,
in
main()
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1492,
in main
parser.parse(filename)
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1351,
in parse
format = self.detect_format(file)
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1306,
in detect_format
format = candidate_format.check_format(file)
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 151, i
n check_format
return RegexFormat(‘iis’, ’ '.join(full_regex), ‘%Y-%m-%d %H:%M:%S’)
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 108, i
n init
self.regex = re.compile(regex + ‘\s*$’) # make sure regex includes end of li
ne
File “C:\Apps\Python\lib\re.py”, line 190, in compile
return _compile(pattern, flags)
File “C:\Apps\Python\lib\re.py”, line 242, in _compile
raise error, v # invalid expression
sre_constants.error: unbalanced parenthesis

So so far I have not managed to import any logs for one reason or another and this is after a complete clean install again. Its a shame as Piwik look like a nice alternative to my other system but I feel it’s still in it’s early days as I have not managed to get this to work at all!

Thanks

Ed


(Matthieu Aubry) #2

You dont need to create a new post. Piwik log import works for hundreds of users so it works. But probably you have found a bug that we’d like to fix.


#3

Sorry thought that this was a different problem as this crashes trying to detect IIS log format. The other post is for NSCA extended format which doesn’t crash like above but doesn’t import proper as other thread. Hence the two different posts…


#4

Have sent file to Fabian regarding this error. So you can reproduce it.

Thanks

Ed


#5

Any news on this one?

Thanks

Ed


#6

:o


(Matthieu Aubry) #7

it’s fixed in: http://builds.piwik.org/piwik-1.12-b3.zip


#8

Hi,

Thanks but unfortantly this hasn’t resolved the above issue. :S Here another dump using the latest build.

Traceback (most recent call last):
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1560,
in
main()
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1524,
in main
parser.parse(filename)
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1378,
in parse
format = self.detect_format(file)
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 1325,
in detect_format
match = candidate_format.check_format(file)
File “W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”, line 160, i
n check_format
self.regex = re.compile(’ '.join(full_regex))
File “C:\Apps\Python\lib\re.py”, line 190, in compile
return _compile(pattern, flags)
File “C:\Apps\Python\lib\re.py”, line 242, in _compile
raise error, v # invalid expression
sre_constants.error: unbalanced parenthesis

Thanks

Ed


#9

Hmm i cant see anything at all to immediately try…

“W:\wwwroot\Private\Piwik\misc\log-analytics\import_logs.py”

could you change it to

“W:\wwwroot\Private\Piwik\misc\loganalytics\importlogs.py” i want to eliminate any potential format errors.

The permissions on this folder are they the same as

“C:\Apps\Python\lib\re.py

Is it possible there is a temp folder that doesnt have the same permissions?


(Matthieu Aubry) #10

Can you add a test in: https://github.com/piwik/piwik/tree/master/misc/log-analytics/tests/logs

that shows a failure when running the tests https://github.com/piwik/piwik/tree/master/misc/log-analytics/tests

then we can try fix it


#11

Have added this to the github, under tests as ‘IIS Import issue’.


#12

Hi,

Can you let me know if you have received the logs correctly in the github ?? I can now longer see them myself now ?

Thanks

Ed


#13

Posted example log at Github


(Matthieu Aubry) #14

Did you try running tests and are they failing with your log file ?


#15

Sorry - not sure I know what you mean ? Do I download the tests directory and do something ??

Thanks

Ed


#16

:o


(Matthieu Aubry) #17

Yes you could try run the tests. But if you don’t know that’s also OK. Instead please create a ticket at dev.piwik.org and attach the failing log file, and the output you get from running the script. We’ll fix it sometimes next few months. Thanks!


(Matthieu Aubry) #18

Maybe it is the same bug as: Import_Logs.py - MySQL error when trying to import logs out of date sequence · Issue #3822 · matomo-org/matomo · GitHub ?


#19

No unfortantly not, I haven’t been able to import 1 single log from either a IIS or a Linux NCSA format. So, so far I’ve not seen Piwik work ;(

This is odd as you must have hundreds of customers whom use it. I haven’t done anything sepecifal to the logs or the servers regarding the logs. And the two servers are different platforms in different parts of the country, I would expect to at least get something to work.

It therefore seems you have very limited people using the import feature or there is somethings fundimentally incorrect in the code.

Then there is the question as to how acturate therefore the import is for those it works for I guess?

Thanks

Ed


#20

Mako you mention you are importing from 2 servers. Is python version the same on both?

How are you connecting to the W drive just a shared volume? What connection type is it? Can you change to another connection type for that shared volume?