Hello,
I am trying to get Piwik running and have been disappointed so far. First the stats:
DB Server:
MySQL
5 GB Ram
2 Intel Xeon 2.4 Ghz (HyperV VM)
Piwik Server:
Windows 2012 R2/IIS
PHP w/WinCache & FastCGI 5.5.11
8 GB RAM
4 Intel Xeon (HyperV VM)
I am trying to bulk import months of IIS logs. I had run archiving previously for some of that, but deleted all the archive tables to start fresh after everything is imported, so archiving is turned off and the scheduled task is disabled.
I wrote a Powershell wrapper to basically iterate through the folders and call the python script import_logs.py with the proper commands (with 4 recorders). What happens is it seems to start off fast enough. Here is a current session:
50855 lines parsed, 47200 lines recorded, 259 records/sec (avg), 670 records/sec (current)
50855 lines parsed, 47510 lines recorded, 259 records/sec (avg), 310 records/sec (current)
51676 lines parsed, 48000 lines recorded, 261 records/sec (avg), 490 records/sec (current)
51676 lines parsed, 48161 lines recorded, 260 records/sec (avg), 161 records/sec (current)
51676 lines parsed, 48382 lines recorded, 260 records/sec (avg), 221 records/sec (current)
51676 lines parsed, 48553 lines recorded, 259 records/sec (avg), 171 records/sec (current)
52489 lines parsed, 48967 lines recorded, 260 records/sec (avg), 414 records/sec (current)
52489 lines parsed, 48967 lines recorded, 259 records/sec (avg), 0 records/sec (current)
52489 lines parsed, 49547 lines recorded, 260 records/sec (avg), 580 records/sec (current)
53312 lines parsed, 49988 lines recorded, 261 records/sec (avg), 441 records/sec (current)
53312 lines parsed, 49988 lines recorded, 260 records/sec (avg), 0 records/sec (current)
53312 lines parsed, 50115 lines recorded, 259 records/sec (avg), 127 records/sec (current)
53312 lines parsed, 50447 lines recorded, 260 records/sec (avg), 332 records/sec (current)
54129 lines parsed, 50836 lines recorded, 260 records/sec (avg), 389 records/sec (current)
54129 lines parsed, 50836 lines recorded, 259 records/sec (avg), 0 records/sec (current)
54129 lines parsed, 50836 lines recorded, 258 records/sec (avg), 0 records/sec (current)
54129 lines parsed, 50985 lines recorded, 257 records/sec (avg), 149 records/sec (current)
54129 lines parsed, 51530 lines recorded, 258 records/sec (avg), 545 records/sec (current)
54940 lines parsed, 51753 lines recorded, 258 records/sec (avg), 223 records/sec (current)
54940 lines parsed, 51888 lines recorded, 258 records/sec (avg), 135 records/sec (current)
54940 lines parsed, 52239 lines recorded, 258 records/sec (avg), 351 records/sec (current)
54940 lines parsed, 52421 lines recorded, 258 records/sec (avg), 182 records/sec (current)
And on the MYSql side it shows 1851 queries/second, 40.6% InnoDB usage (I have it set to 2 G I believe), and 6 Innodb writers/second.
What seems to happen is the qps drops to almost nothing as does the reads/writes. The log from the python script starts to drop off and decrease as well until eventually I get a Fatal Error. Below is an example. This happens after processing logs for awhile…maybe after an hour or two usually. We need to be able to import all of these logs and I don’t understand why piwik is dying. I also usually can’t access the dashboard when the fatal error starts…the site hangs.
95397 lines parsed, 65369 lines recorded, 157 records/sec (avg), 0 records/sec (current)
95397 lines parsed, 65369 lines recorded, 156 records/sec (avg), 0 records/sec (current)
95397 lines parsed, 65369 lines recorded, 156 records/sec (avg), 0 records/sec (current)
95397 lines parsed, 65369 lines recorded, 155 records/sec (avg), 0 records/sec (current)
95397 lines parsed, 65369 lines recorded, 155 records/sec (avg), 0 records/sec (current)
96031 lines parsed, 66128 lines recorded, 157 records/sec (avg), 759 records/sec (current)
96252 lines parsed, 66202 lines recorded, 156 records/sec (avg), 74 records/sec (current)
96252 lines parsed, 66202 lines recorded, 156 records/sec (avg), 0 records/sec (current)
96252 lines parsed, 66202 lines recorded, 156 records/sec (avg), 0 records/sec (current)
96252 lines parsed, 66202 lines recorded, 155 records/sec (avg), 0 records/sec (current)
96252 lines parsed, 66202 lines recorded, 155 records/sec (avg), 0 records/sec (current)
96252 lines parsed, 66202 lines recorded, 154 records/sec (avg), 0 records/sec (current)
97092 lines parsed, 66741 lines recorded, 155 records/sec (avg), 539 records/sec (current)
97092 lines parsed, 66769 lines recorded, 155 records/sec (avg), 28 records/sec (current)
97092 lines parsed, 66769 lines recorded, 155 records/sec (avg), 0 records/sec (current)
97092 lines parsed, 66769 lines recorded, 154 records/sec (avg), 0 records/sec (current)
97092 lines parsed, 66769 lines recorded, 154 records/sec (avg), 0 records/sec (current)
97092 lines parsed, 66769 lines recorded, 154 records/sec (avg), 0 records/sec (current)
97092 lines parsed, 66769 lines recorded, 153 records/sec (avg), 0 records/sec (current)
97956 lines parsed, 67330 lines recorded, 154 records/sec (avg), 561 records/sec (current)
97956 lines parsed, 67393 lines recorded, 154 records/sec (avg), 63 records/sec (current)
97956 lines parsed, 67393 lines recorded, 154 records/sec (avg), 0 records/sec (current)
97956 lines parsed, 67393 lines recorded, 153 records/sec (avg), 0 records/sec (current)
97956 lines parsed, 67393 lines recorded, 153 records/sec (avg), 0 records/sec (current)
97956 lines parsed, 67393 lines recorded, 153 records/sec (avg), 0 records/sec (current)
97956 lines parsed, 67393 lines recorded, 152 records/sec (avg), 0 records/sec (current)
97956 lines parsed, 67393 lines recorded, 152 records/sec (avg), 0 records/sec (current)
97956 lines parsed, 67393 lines recorded, 151 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68138 lines recorded, 153 records/sec (avg), 745 records/sec (current)
98805 lines parsed, 68187 lines recorded, 153 records/sec (avg), 49 records/sec (current)
98805 lines parsed, 68187 lines recorded, 152 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68187 lines recorded, 152 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68187 lines recorded, 152 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68187 lines recorded, 151 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68187 lines recorded, 151 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68187 lines recorded, 150 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68187 lines recorded, 150 records/sec (avg), 0 records/sec (current)
98805 lines parsed, 68187 lines recorded, 150 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 151 records/sec (avg), 775 records/sec (current)
99659 lines parsed, 68962 lines recorded, 151 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 150 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 150 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 150 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 149 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 149 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 149 records/sec (avg), 0 records/sec (current)
99659 lines parsed, 68962 lines recorded, 149 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69698 lines recorded, 150 records/sec (avg), 736 records/sec (current)
100511 lines parsed, 69707 lines recorded, 149 records/sec (avg), 9 records/sec (current)
100511 lines parsed, 69707 lines recorded, 149 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69707 lines recorded, 149 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69707 lines recorded, 148 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69707 lines recorded, 148 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69707 lines recorded, 148 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69707 lines recorded, 148 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69707 lines recorded, 147 records/sec (avg), 0 records/sec (current)
100511 lines parsed, 69707 lines recorded, 147 records/sec (avg), 0 records/sec (current)
100929 lines parsed, 70455 lines recorded, 148 records/sec (avg), 748 records/sec (current)
101350 lines parsed, 70505 lines recorded, 148 records/sec (avg), 50 records/sec (current)
101350 lines parsed, 70505 lines recorded, 148 records/sec (avg), 0 records/sec (current)
101350 lines parsed, 70505 lines recorded, 147 records/sec (avg), 0 records/sec (current)
101350 lines parsed, 70505 lines recorded, 147 records/sec (avg), 0 records/sec (current)
101350 lines parsed, 70505 lines recorded, 147 records/sec (avg), 0 records/sec (current)
101350 lines parsed, 70505 lines recorded, 146 records/sec (avg), 0 records/sec (current)
101350 lines parsed, 70505 lines recorded, 146 records/sec (avg), 0 records/sec (current)
101350 lines parsed, 70505 lines recorded, 146 records/sec (avg), 0 records/sec (current)
101350 lines parsed, 70505 lines recorded, 145 records/sec (avg), 0 records/sec (current)
101775 lines parsed, 71263 lines recorded, 147 records/sec (avg), 758 records/sec (current)
102286 lines parsed, 71284 lines recorded, 146 records/sec (avg), 21 records/sec (current)
102286 lines parsed, 71342 lines recorded, 146 records/sec (avg), 58 records/sec (current)
102286 lines parsed, 71342 lines recorded, 146 records/sec (avg), 0 records/sec (current)
102286 lines parsed, 71342 lines recorded, 146 records/sec (avg), 0 records/sec (current)
102286 lines parsed, 71342 lines recorded, 145 records/sec (avg), 0 records/sec (current)
102286 lines parsed, 71342 lines recorded, 145 records/sec (avg), 0 records/sec (current)
102286 lines parsed, 71342 lines recorded, 145 records/sec (avg), 0 records/sec (current)
102286 lines parsed, 71342 lines recorded, 144 records/sec (avg), 0 records/sec (current)
102286 lines parsed, 71342 lines recorded, 144 records/sec (avg), 0 records/sec (current)
103449 lines parsed, 72132 lines recorded, 145 records/sec (avg), 790 records/sec (current)
103659 lines parsed, 72148 lines recorded, 145 records/sec (avg), 16 records/sec (current)
103659 lines parsed, 72148 lines recorded, 145 records/sec (avg), 0 records/sec (current)
103659 lines parsed, 72297 lines recorded, 145 records/sec (avg), 149 records/sec (current)
103659 lines parsed, 72297 lines recorded, 145 records/sec (avg), 0 records/sec (current)
103659 lines parsed, 72297 lines recorded, 144 records/sec (avg), 0 records/sec (current)
103659 lines parsed, 72297 lines recorded, 144 records/sec (avg), 0 records/sec (current)
103659 lines parsed, 72297 lines recorded, 144 records/sec (avg), 0 records/sec (current)
103659 lines parsed, 72297 lines recorded, 143 records/sec (avg), 0 records/sec (current)
103659 lines parsed, 72297 lines recorded, 143 records/sec (avg), 0 records/sec (current)
104920 lines parsed, 73064 lines recorded, 144 records/sec (avg), 767 records/sec (current)
104920 lines parsed, 73064 lines recorded, 144 records/sec (avg), 0 records/sec (current)
104920 lines parsed, 73199 lines recorded, 144 records/sec (avg), 135 records/sec (current)
104920 lines parsed, 73361 lines recorded, 144 records/sec (avg), 162 records/sec (current)
104920 lines parsed, 73603 lines recorded, 144 records/sec (avg), 242 records/sec (current)
104920 lines parsed, 73603 lines recorded, 144 records/sec (avg), 0 records/sec (current)
104920 lines parsed, 73603 lines recorded, 144 records/sec (avg), 0 records/sec (current)
104920 lines parsed, 73603 lines recorded, 143 records/sec (avg), 0 records/sec (current)
104920 lines parsed, 73603 lines recorded, 143 records/sec (avg), 0 records/sec (current)
104920 lines parsed, 73603 lines recorded, 143 records/sec (avg), 0 records/sec (current)
106333 lines parsed, 74365 lines recorded, 144 records/sec (avg), 762 records/sec (current)
106333 lines parsed, 74365 lines recorded, 144 records/sec (avg), 0 records/sec (current)
106333 lines parsed, 74684 lines recorded, 144 records/sec (avg), 319 records/sec (current)
106333 lines parsed, 74684 lines recorded, 144 records/sec (avg), 0 records/sec (current)
106333 lines parsed, 74684 lines recorded, 143 records/sec (avg), 0 records/sec (current)
106333 lines parsed, 75039 lines recorded, 144 records/sec (avg), 355 records/sec (current)
106333 lines parsed, 75039 lines recorded, 144 records/sec (avg), 0 records/sec (current)
106333 lines parsed, 75039 lines recorded, 143 records/sec (avg), 0 records/sec (current)
106333 lines parsed, 75039 lines recorded, 143 records/sec (avg), 0 records/sec (current)
107747 lines parsed, 75762 lines recorded, 144 records/sec (avg), 723 records/sec (current)
107747 lines parsed, 75762 lines recorded, 144 records/sec (avg), 0 records/sec (current)
107747 lines parsed, 75762 lines recorded, 144 records/sec (avg), 0 records/sec (current)
107747 lines parsed, 76191 lines recorded, 144 records/sec (avg), 429 records/sec (current)
109066 lines parsed, 76706 lines recorded, 145 records/sec (avg), 515 records/sec (current)
109066 lines parsed, 76706 lines recorded, 145 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 144 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 144 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 144 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 143 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 143 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 143 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 143 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 142 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 142 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 142 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 142 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 141 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 141 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 141 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 140 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 140 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 140 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 140 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 139 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 139 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 139 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 139 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 138 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 138 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 138 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 138 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 137 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 137 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 137 records/sec (avg), 0 records/sec (current)
109066 lines parsed, 76706 lines recorded, 137 records/sec (avg), 0 records/sec (current)
110370 lines parsed, 76831 lines recorded, 137 records/sec (avg), 125 records/sec (current)
110370 lines parsed, 76831 lines recorded, 136 records/sec (avg), 0 records/sec (current)
110370 lines parsed, 76831 lines recorded, 136 records/sec (avg), 0 records/sec (current)
110370 lines parsed, 76831 lines recorded, 136 records/sec (avg), 0 records/sec (current)
Fatal error: Internal Server Error