Hi, I have a weird problem where the log imports on one of our sites over night has slowed down from around 450-600 records/sec to about 9-10 records/sec on average.
We are in the process of migrating from a single physical server to a two server setup with separate web and database servers. The new setup is still in testing phase but we are seeing the same on both setups which makes me think that there is something with the logs causing this. Other sites that are importing logs on the same servers are not having the same problem. From what I can tell it started on the 30th where the import seem to start at normal speed and the halt. I did get an error on the new server. Pasting the logs below.
Has anyone experienced anything like this before? I have looked at the logs but as far as I can see there aren’t any large differences.
Best regards
Victor
Parsing log /opt/san/logs/<webserver>/u_ex201130.log...
229011 lines parsed, 80336 lines recorded, 669 records/sec (avg), 669 records/sec (current)
236826 lines parsed, 83709 lines recorded, 348 records/sec (avg), 28 records/sec (current)
238593 lines parsed, 84574 lines recorded, 234 records/sec (avg), 7 records/sec (current)
Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 805, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 758, in run
self.__target(*self.__args, **self.__kwargs)
File "/opt/san/matomo/misc/log-analytics/import_logs.py", line 1871, in _run_bulk
self._record_hits(hits)
File "/opt/san/matomo/misc/log-analytics/import_logs.py", line 2029, in _record_hits
on_failure=self._on_tracking_failure
File "/opt/san/matomo/misc/log-analytics/import_logs.py", line 1651, in call
data=data)
File "/opt/san/matomo/misc/log-analytics/import_logs.py", line 1605, in _call_wrapper
response = func(*args, **kwargs)
File "/opt/san/matomo/misc/log-analytics/import_logs.py", line 1541, in _call
response = opener.open(request, timeout = timeout)
File "/usr/lib64/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 1243, in https_open
context=self._context)
File "/usr/lib64/python2.7/urllib2.py", line 1203, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib64/python2.7/httplib.py", line 1148, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 448, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 404, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib64/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib64/python2.7/ssl.py", line 754, in recv
return self.read(buflen)
File "/usr/lib64/python2.7/ssl.py", line 641, in read
v = self._sslobj.read(len)
SSLError: ('The read operation timed out',)
238593 lines parsed, 84882 lines recorded, 176 records/sec (avg), 2 records/sec (current)
238593 lines parsed, 85182 lines recorded, 141 records/sec (avg), 2 records/sec (current)
238593 lines parsed, 85815 lines recorded, 119 records/sec (avg), 5 records/sec (current)
238593 lines parsed, 86445 lines recorded, 102 records/sec (avg), 5 records/sec (current)
238593 lines parsed, 86991 lines recorded, 90 records/sec (avg), 4 records/sec (current)
238593 lines parsed, 87580 lines recorded, 81 records/sec (avg), 4 records/sec (current)
238593 lines parsed, 87580 lines recorded, 72 records/sec (avg), 0 records/sec (current)
238593 lines parsed, 87580 lines recorded, 66 records/sec (avg), 0 records/sec (current)
238593 lines parsed, 87580 lines recorded, 60 records/sec (avg), 0 records/sec (current)
238593 lines parsed, 87580 lines recorded, 56 records/sec (avg), 0 records/sec (current)
238593 lines parsed, 87580 lines recorded, 52 records/sec (avg), 0 records/sec (current)