Import_logs.py not working as expected after major upgrade

Support & Bugs
1

We upgrade to:

  • Debian 10.1
  • PHP 7.3
  • Matomo 3.11.0
  • Mariadb 10.3
  • Python 2.7.16
  • Apache 2.4.38

And now import_logs.py is not working anymore. We are importing logs from apache. During import_logs.py it crashes and freezes.
We get the following error from import_logs.py :

out 06 15:00:15 server systemd[1]: Started Rotate log files.
out 07 00:00:01 server systemd[1]: Starting Rotate log files...
out 07 00:05:19 server logrotate[23463]: Exception in thread Thread-3:
out 07 00:05:19 server logrotate[23463]: Traceback (most recent call last):
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
out 07 00:05:19 server logrotate[23463]:     self.run()
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/threading.py", line 754, in run
out 07 00:05:19 server logrotate[23463]:     self.__target(*self.__args, **self.__kwargs)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1806, in _run_bulk
out 07 00:05:19 server logrotate[23463]:     self._record_hits(hits)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1964, in _record_hits
out 07 00:05:19 server logrotate[23463]:     on_failure=self._on_tracking_failure
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1586, in call
out 07 00:05:19 server logrotate[23463]:     data=data)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1540, in _call_wrapper
out 07 00:05:19 server logrotate[23463]:     response = func(*args, **kwargs)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1476, in _call
out 07 00:05:19 server logrotate[23463]:     response = opener.open(request, timeout = timeout)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 429, in open
out 07 00:05:19 server logrotate[23463]:     response = self._open(req, data)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
out 07 00:05:19 server logrotate[23463]:     '_open', req)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
out 07 00:05:19 server logrotate[23463]:     result = func(*args)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
out 07 00:05:19 server logrotate[23463]:     context=self._context)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
out 07 00:05:19 server logrotate[23463]:     r = h.getresponse(buffering=True)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
out 07 00:05:19 server logrotate[23463]:     response.begin()
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/httplib.py", line 438, in begin
out 07 00:05:19 server logrotate[23463]:     version, status, reason = self._read_status()
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
out 07 00:05:19 server logrotate[23463]:     line = self.fp.readline(_MAXLINE + 1)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/socket.py", line 480, in readline
out 07 00:05:19 server logrotate[23463]:     data = self._sock.recv(self._rbufsize)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/ssl.py", line 754, in recv
out 07 00:05:19 server logrotate[23463]:     return self.read(buflen)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/ssl.py", line 641, in read
out 07 00:05:19 server logrotate[23463]:     v = self._sslobj.read(len)
out 07 00:05:19 server logrotate[23463]: SSLError: ('The read operation timed out',)
out 07 00:05:19 server logrotate[23463]: Exception in thread Thread-4:
out 07 00:05:19 server logrotate[23463]: Traceback (most recent call last):
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
out 07 00:05:19 server logrotate[23463]:     self.run()
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/threading.py", line 754, in run
out 07 00:05:19 server logrotate[23463]:     self.__target(*self.__args, **self.__kwargs)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1806, in _run_bulk
out 07 00:05:19 server logrotate[23463]:     self._record_hits(hits)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1964, in _record_hits
out 07 00:05:19 server logrotate[23463]:     on_failure=self._on_tracking_failure
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1586, in call
out 07 00:05:19 server logrotate[23463]:     data=data)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1540, in _call_wrapper
out 07 00:05:19 server logrotate[23463]:     response = func(*args, **kwargs)
out 07 00:05:19 server logrotate[23463]:   File "/var/www/matomo/misc/log-analytics/import_logs.py", line 1476, in _call
out 07 00:05:19 server logrotate[23463]:     response = opener.open(request, timeout = timeout)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 429, in open
out 07 00:05:19 server logrotate[23463]:     response = self._open(req, data)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
out 07 00:05:19 server logrotate[23463]:     '_open', req)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
out 07 00:05:19 server logrotate[23463]:     result = func(*args)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
out 07 00:05:19 server logrotate[23463]:     context=self._context)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
out 07 00:05:19 server logrotate[23463]:     r = h.getresponse(buffering=True)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
out 07 00:05:19 server logrotate[23463]:     response.begin()
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/httplib.py", line 438, in begin
out 07 00:05:19 server logrotate[23463]:     version, status, reason = self._read_status()
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
out 07 00:05:19 server logrotate[23463]:     line = self.fp.readline(_MAXLINE + 1)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/socket.py", line 480, in readline
out 07 00:05:19 server logrotate[23463]:     data = self._sock.recv(self._rbufsize)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/ssl.py", line 754, in recv
out 07 00:05:19 server logrotate[23463]:     return self.read(buflen)
out 07 00:05:19 server logrotate[23463]:   File "/usr/lib/python2.7/ssl.py", line 641, in read
out 07 00:05:19 server logrotate[23463]:     v = self._sslobj.read(len)
out 07 00:05:19 server logrotate[23463]: SSLError: ('The read operation timed out',)
out 07 08:54:59 server logrotate[23463]: error: error running shared prerotate script for '/var/log/apache2/*.log '
out 07 08:55:00 server systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
out 07 08:55:00 server systemd[1]: logrotate.service: Failed with result 'exit-code'.

What can be done?

2

Hi,

It seems like the python script tries to send data to Matomo

You might want to test my new beta import script that supports python3 (matomo-log-analytics/import_logs.py at 1753845e1a8ef7d803074201a027190d557ed924 · matomo-org/matomo-log-analytics · GitHub) (https://github.com/matomo-org/matomo-log-analytics/pull/242)

Apart from this check your webserver/php/matomo error log for errors.

3

This new import_logs.py version only supports Python 3.5, 3.6 or 3.7 but I am still using Python 2.7

4

Well this is kind of the point of my work to migrate the import script to Python3 (after all python2 support ends very soon). But no worries, the old script will still be supported at least until Matomo4.

5

Thank you for replying. I am using the import script on cold apache log files (logrotate postrotate scripts) now and it is working so far. I used to run import script on hot apache log files (logrotate prerotate scripts) and it was crashing during execution.