Log-analytics stops recording after a certain number of lines


(O. Herbst) #1

Hello,

We want to read very large logs (approx. 20GB - 30GB) using log analytics and the Python script.
The format is recognized correctly, but only the first about 20,000 records are taken over. Then all other log lines are recognized as invalid. However, the lines are definitely in the same format as the previous ones.
Splitting the data into smaller files does not help either. Is there a limitation in Python or in the script or in Matomo?

Here the output:
14449 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 1400 lines recorded, 199 records/sec (avg), 1400 records/sec (current)
20000 lines parsed, 1400 lines recorded, 174 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 1400 lines recorded, 155 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 1400 lines recorded, 139 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 1400 lines recorded, 127 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 1400 lines recorded, 116 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 2800 lines recorded, 215 records/sec (avg), 1400 records/sec (current)
20000 lines parsed, 2800 lines recorded, 199 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 2800 lines recorded, 186 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 2800 lines recorded, 174 records/sec (avg), 0 records/sec (current)
20000 lines parsed, 2800 lines recorded, 164 records/sec (avg), 0 records/sec (current)

Logs import summary

4160 requests imported successfully
0 requests were downloads
15840 requests ignored:
    0 HTTP errors
    0 HTTP redirects
    15840 invalid log lines
    0 filtered log lines
    0 requests did not match any known site
    0 requests did not match any --hostname
    0 requests done by bots, search engines...
    0 requests to static resources (css, js, images, ico, ttf...)
    0 requests to file downloads did not match any --download-extensions

Website import summary

4160 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

Thanks for the Help
regards