Upgrade from 1.8.2 to 2.3.0 - Archiving very slow


(andrewdrake) #1

Hi Guys,

I have around 8000 sites, most low traffic, ie <20 visitors/day.

The arching used to take 2-3 hours, which was fine as not a high spec server and I set it to run evyer 6 hours.

Now the archiving has been running for about 5 days, have had to upgrade memory on the server and stop it a few times as it was using swap.

Any reason why or how this could happen?

A lot of slow queries also, this one for example:
SELECT idarchive FROM piwik_archive_numeric_2012_01 WHERE name = ‘locked_6d4d2a4c1c2fd76812d56193b99906d6’ LIMIT 1

Andrew


(Matthieu Aubry) #2

please upgrade to 2.4.0 beta: I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo
does it work better?


(andrewdrake) #3

The archiving is still running, is it bad to stop it while running? Happy to stop and and upgrade, otherwise I think it’s got another couple of weeks before it finishes.

Andrew


(Matthieu Aubry) #4

it’s OK and safe to stop archiving while it’s running.


(andrewdrake) #5

Ok great I’ll give the 2.4 beta version a go


(andrewdrake) #6

I have now upgraded to latest Beta (2.4.0-b8).

Running the archiving via console again, still seems very slow. This is how long it took to do 1 site, running the console archive command manually via SSH.

INFO CoreConsole[2014-07-01 07:25:05] [fa2ff] Archived website id = 942, period = day, 0 visits in last 52 days, 0 visits today, Time elapsed: 32.338s
INFO CoreConsole[2014-07-01 07:26:12] [fa2ff] Archived website id = 942, period = week, 0 visits in last 25 weeks, 0 visits this week, Time elapsed: 66.787s
INFO CoreConsole[2014-07-01 07:30:15] [fa2ff] Archived website id = 942, period = month, 4 visits in last 25 months, 0 visits this month, Time elapsed: 243.899s
INFO CoreConsole[2014-07-01 07:32:25] [fa2ff] Archived website id = 942, period = year, 8 visits in last 7 years, 0 visits this year, Time elapsed: 129.397s
INFO CoreConsole[2014-07-01 07:32:25] [fa2ff] Archived website id = 942, 4 API requests, Time elapsed: 473.525s [1/7698 done]

Before, with Version 1.8.2 it used to run at this speed:

Archived website id = 28, period = day, Time elapsed: 0.365s
Archived website id = 28, period = week, 62 visits, Time elapsed: 0.515s
Archived website id = 28, period = month, 319 visits, Time elapsed: 0.175s
Archived website id = 28, period = year, 4421 visits, Time elapsed: 0.349s


(Matthieu Aubry) #7

you cant’ really compare run times like this because it may be different (eg. the 0.3s archiving was likely only reading existing archived data, when the 30s archiving call is processing the data)


(andrewdrake) #8

Ok.

It seems to be a little faster now, still slow though. Coming up on 24 hours and is at this stage. At this rate it will take 4 days to complete, with 2.3.0 it was going to take weeks.

INFO CoreConsole[2014-07-02 04:43:15] [fa2ff] Archived website id = 2709, period = day, 0 visits in last 19 days, 0 visits today, Time elapsed: 12.333s
INFO CoreConsole[2014-07-02 04:44:14] [fa2ff] Archived website id = 2709, period = week, 0 visits in last 19 weeks, 0 visits this week, Time elapsed: 59.342s
INFO CoreConsole[2014-07-02 04:44:20] [fa2ff] Archived website id = 2709, period = month, 9 visits in last 19 months, 0 visits this month, Time elapsed: 5.453s
INFO CoreConsole[2014-07-02 04:44:21] [fa2ff] Archived website id = 2709, period = year, 38 visits in last 7 years, 0 visits this year, Time elapsed: 1.368s
INFO CoreConsole[2014-07-02 04:44:21] [fa2ff] Archived website id = 2709, 4 API requests, Time elapsed: 78.672s [1757/7698 done]


(Matthieu Aubry) #9

can you try with 2.4.0-rc as we made further improvements: I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo


(andrewdrake) #10

Yes, can do. I’ll upgrade tomorrow and let you know how it goes.

PS. Do you guys have a hosted version available? I’d be happy to look at options rather than hosting myself.


(Matthieu Aubry) #11

Sure, see: Cloud-Hosted Piwik - Analytics Platform - Matomo


(andrewdrake) #12

Just upgraded top 2.4.0, seems to be running the archiving much faster now. Will post details here once complete.