New install using access logs. Import and archiving very slow

Support & Bugs
1

Hi All,

I’ve setup a piwik box for processing access logs.

These access logs are around 6-7million lines and take very very long to process (I’m getting ~200-300 records/sec).

This isn’t too bad but then the archive task takes even longer.

I’m not even sure that piwik can actually import these records faster than they get generated.

Anyway to speed this up?

Also I’m using page url contains for segments, I guess I could speed this up by using an is match rather than contains - is there an variable that contains the request without GET params as that is the only reason I’m using contains and not is (e.g. segment to match request.php ignoring GET params)?

2

Do you mean 6-7 millions lines (i.e. page views) per day?

Have a look here for a rough estimate of what can be done with Piwik: Optimize and Scale Piwik - Analytics Platform - Matomo