import_logs.py - Error when connecting to Piwik

Support & Bugs
1

Hello,
I Have a fresh install from Piwik 1.8.4 on a Win Server 2008 with IIS 7.5.
Tried to import a logfile with the “import_logs.py” script, but failed with “Error when connecting to Piwik”.


D:\test\import_logs.py --debug --url=http://piwik.example/live/ iwd.201104.log --login=#### --password=#### --idsite=2 --recorders=1 

2012-10-10 12:48:36,171: [DEBUG] Accepted hostnames: all
2012-10-10 12:48:36,171: [DEBUG] Piwik URL is: http://piwik.example.com/live/
2012-10-10 12:48:36,171: [DEBUG] No token-auth specified
2012-10-10 12:48:36,171: [DEBUG] Using credentials: (login = #####, password = #####)
2012-10-10 12:48:36,390: [DEBUG] Authentication token token_auth is: #####
2012-10-10 12:48:36,390: [DEBUG] Resolver: static
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2012-10-10 12:48:36,592: [DEBUG] Launched recorder
Parsing log iwd.201104.log...
2012-10-10 12:48:36,592: [DEBUG] Detecting the log format
2012-10-10 12:48:36,592: [DEBUG] Format ncsa_extended does not match
2012-10-10 12:48:36,592: [DEBUG] Format iis does not match
2012-10-10 12:48:36,592: [DEBUG] Format common_complete does not match
2012-10-10 12:48:36,592: [DEBUG] Format common matches
4180 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
8308 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2012-10-10 12:48:38,967: [DEBUG] Error when connecting to Piwik: HTTP Error 404: Not Found
12245 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
16021 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2012-10-10 12:48:40,969: [DEBUG] Error when connecting to Piwik: HTTP Error 404: Not Found
19946 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
23333 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2012-10-10 12:48:42,969: [DEBUG] Error when connecting to Piwik: HTTP Error 404: Not Found
Fatal error: Not Found
You can restart the import of "iwd.201104.log" from the point it failed by specifying --skip=0 on the command line.

here are the entries form the server log


2012-10-10 10:48:36 172.18.7.222 POST /live/ - 80 - 172.18.7.222 Python-urllib/2.7 200 0 0 203
2012-10-10 10:48:36 172.18.7.222 POST /live/ - 80 - 172.18.7.222 Python-urllib/2.7 200 0 0 203
2012-10-10 10:48:38 172.18.7.222 POST /live/piwik.php - 80 - 172.18.7.222 Python-urllib/2.7 404 0 2 0
2012-10-10 10:48:40 172.18.7.222 POST /live/piwik.php - 80 - 172.18.7.222 Python-urllib/2.7 404 0 2 0
2012-10-10 10:48:42 172.18.7.222 POST /live/piwik.php - 80 - 172.18.7.222 Python-urllib/2.7 404 0 2 0

it seems user “jay_haitch” have the same problem, see:
Forum Thread: import_logs.py fails contacting Piwik served from Nginx

so I changed line 764 of the “import_logs.py” script to "raise logging.debug(‘Error when connecting to Piwik: %s’, e)"
like “cyril” requested, here is the output:


2012-10-10 13:31:56,415: [DEBUG] Error when connecting to Piwik: HTTP Error 404:
 Not Found
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Py\lib\threading.py", line 551, in __bootstrap_inner
    self.run()
  File "C:\Py\lib\threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "D:\test\import_logs.py", line 968, in _run
    self._record_hits(hits)
  File "D:\test\import_logs.py", line 1049, in _record_hits
    on_failure=self._on_tracking_failure
  File "D:\test\import_logs.py", line 780, in call
    data=data)
  File "D:\test\import_logs.py", line 764, in _call_wrapper
    raise logging.debug('Error when connecting to Piwik: %s', e)
TypeError: exceptions must be old-style classes or derived from BaseException, n
ot NoneType

14 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)

sadly no change after manually adding the column “location_geoip_region” to the table “webstat_log_visit” as described in this post.

any help appreciated, please advice.

2

Error returns HTTP Error 404: Not Found

Look in the error file

3

embarrassing. it was only a configuration issue on the server… tnx for yout hint.

4

Hi,

I’m having the same problem but the error is rather than HTTP 404 ??? Any ideas?

Thanks

Ed

5

Look in the server error log file ?

6

Hi,

The server error log ? Is this specific to Piwik or Python ??

Thanks

Ed

7

If you mean the IIS logs.Its similar to bbkfp post except that I don’t have 404 I have 200 all the way down, which means OK. So bit stumped…

8

Tried changing the IP and website name. Still no go thou… Any ideas? Would be good to see some logs coming in :wink:

Gone through the logs and there fine and they import into our older system ok. They are NCSA extended format. Oddly it did seem to do it very first time around but then it failed with same error, everytime again it has failed as described above.

9

BUMP X(

10

what version of python? Ideal minimum is 2.7.3

11

Hi,

Yes it is version 2.7.3

Thanks

Ed

12

Check this thread below in particular the format of source and location file log.

C:\Python27>python C:\Inetpub\piwik\misc\log-analytics\import_logs.py --url=http://piwik/ D:\IISLOGS\GEMINI\W3SVC1908953133\ex121101.log --i
dsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --dry-run

http://forum.piwik.org/read.php?2,90223

There was a bug earlier that perhaps is still a issue from what i saw there regarding dates of imported files.

13

Hi,

Yes this seems to fit! If I move forward in time it runs, however I’ve not managed to get it to run a full log yet. It seems to stop with a Fatal Error: None problem. The debug shows the message 500 Internal Server Error a few lines before hitting a Piwik urlopen error. Then the log wont run again. Nor will logs that are older.

Thanks

Ed

14

Post some output if you can may help source the problem…

15

Output is as follows:

2013-03-07 21:35:02,334: [DEBUG] Resolver: static
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2013-03-07 21:35:02,552: [DEBUG] Launched recorder
Parsing log log-Feb-2009…
2013-03-07 21:35:02,568: [DEBUG] Detecting the log format
2013-03-07 21:35:02,584: [DEBUG] Format ncsa_extended matches
3221 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
2013-03-07 21:35:03,865: [DEBUG] Error when connecting to Piwik:
5071 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
5071 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
5071 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
2013-03-07 21:35:06,818: [DEBUG] Error when connecting to Piwik:
5071 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
5071 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
5071 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
2013-03-07 21:35:10,395: [DEBUG] Error when connecting to Piwik:
Fatal error: None
You can restart the import of “log-Feb-2009” from the point it fail
ed by specifying --skip=79 on the command line.

If I use the parameter --disable-bulk-tracking then it seems to say more about lines recorded but still has the urlopen error problem.

Would love to see this working :S

16

Check your piwik server error logs ?

17

Hi

When you say check server logs. Do you mean the auctual server logs or do you mean a log specific to Piwik ? The server logs have no errors.

Thx

Ed

18

Is the IIS server 32 or 64 bit?

Is the server behind a proxy?

What version of urllib?

19

IIS 32 bit
Its not behind a proxy but behind a firewall it can access DNS for resolution but not access external websites.

Not sure what version of urllib - whatever can with the Piwik or Python 2.7.3 I think …???

20

Have updated to latest version 1.11 and still the same problem :X