High-Traffic experiences


(farmerjohn) #1

I’m experimenting with Piwik for use in a multi-domain, high-traffic production environment. I’ve seen very little discussion of what makes ‘high-traffic’ with regard to Piwik and/or how it performs under pressure.

We serve upwards of 6M page views per day. End-user experience is critical - any noticeable slowdown in site performance is unacceptable.

Any comments that anyone has to further the discussion are appreciated.

UPDATE: see the post: 301 Moved Permanently for more info about scalable high traffic performance piwik


(Bill) #2

hello.how are you.

I am preparing an implementation plan by Piwik. The site scale is million-level pv
every day.
Now,Performance is the focus of attention。Please give suggestions on some performance。Thanks.


(farmerjohn) #3

Well, more apparent ‘issues’ have cropped up:

1 - We run sites in a load-balanced environment. It occurred to me that the installer writes files. Is that limited to /config/config.ini.php and /config/manifest.inc.php?

2 - Writing flat files to the server (./tmp/sessions/, ./tmp/templates_c/) is out of the question. Has anyone developed a work-around for this?

3 - I’d estimate that, with our current traffic patterns (and looking at how many inserts are generated for each ‘visit’), piwik_log_visit/link_visit_action/action tables will grow to somewhere around 20-25M rows for a month. I read elsewhere in the forums that those tables don’t flush. Is that something that we’ll have to implement ourselves?

Thanks in advance,

John


(vipsoft) #4

re: tmp/sessions - that’s a web server configuration issue; in your environment, you’d likely look at changing php’s session handler (default is “files”) to something like memcached; in a future version (post-1.0), we may implement a database-backed store for the session handler


(vipsoft) #5

re: templates_c - I believe the caching of compiled templates can be disabled

Website attributes are cached in tmp/cache/tracker. No easy workaround for this though.


(Matthieu Aubry) #6

1 - installer doesn’t write manifest, it only writes the config.ini.php
2 - I don’t see why writing template files on disk is out of the question? regarding the session files, that is an issue though, and as anthon said, is something that could be adressed (if you would like to contribute, this would make it in piwik much faster for sure)
3 - flushing will be adressed soon, probably in the next 3 months (see http://dev.piwik.org/trac/ticket/5)[/url]. There is a work around in the meantime: [url=http://piwik.org/faq/troubleshooting/#faq_42]http://piwik.org/faq/troubleshooting/#faq_42)


(Matthieu Aubry) #7

I wrote the cache on disk code in such a way that it should allow to easily add a memcache datastore (ie. create a new class that implements the interface). I’m pretty sure this would be a cheap development to add memcache support for cached tracking data (couple days maximum).


(farmerjohn) #8

Our pool servers are Gluster nodes. Their filesystems are read-only.

We do have ten Memcache(d) servers out there… I have a couple of programmers looking at moving sessions to those.

Did anyone have any opinions on how Piwik is going to function with piwik_log_* tables were rows number in the tens of millions?

Thanks,

John


(Matthieu Aubry) #9

[quote=farmerjohn @ May 7 2010, 06:12 PM) <{POST_SNAPBACK}>

We do have ten Memcache(d) servers out there… I have a couple of programmers looking at moving sessions to those.

Nice, let us know your plans regarding this, this should be in Piwik core style_emoticons/<#EMO_DIR#>/smile.gif (ensuring we will maintain it, anyone can benefit, etc.]Did anyone have any opinions on how Piwik is going to function with piwik_log_* tables were rows number in the tens of millions?[/quote]

It is basically down to MySQL performance at this scale. Considering you can safely purge data older than 31 days, this should not be problem in your case. The highest piwik user publicly know is Maciej with 20M pages per month (http://lists.piwik.org/pipermail/piwik-hackers/2010-April/000859.html) to my knowledge, he doesn’t purge the DB yet.


(vipsoft) #10

My definition of easy is somewhere between a one-line fix and something I can code in an hour with one hand, while holding an infant in the other arm. style_emoticons/<#EMO_DIR#>/wink.gif


(farmerjohn) #11

That’s my first LOL of today… having seven kids myself (three under five), I can so feel your pain. Happy Friday!

J


(vipsoft) #12

If you recall, as part of http://dev.piwik.org/trac/ticket/806, I tested Piwik with the session save handler set to files, sqlite, or memcache. These are configred outside of Piwik.

Also, I didn’t test mm (memory mapped) as it doesn’t support Windows, is no longer being maintained, and reportedly has concurrency issues.


(Thomas Seifert) #13

I’m running Piwik also with load-balanced php backends and got no problem with only local storage.
The only problem are the sessions right now which requires me to login twice but just using memcached for session storage might skip this problem.

I know, your requirement is different but just wanted to throw in another possibility.

Regards,

thomas


(Matthieu Aubry) #14

See the post: 301 Moved Permanently


(Matthieu Aubry) #15

See High traffic Piwik FAQ