Segfault when running archive.sh :(


#1

Hi,

Today I upgraded our installation of Piwik from 1.0.0 to 1.2.1. Apart from it taking a long time to upgrade the (4.8GB) database it went smoothly. Until I re-enabled the archiving…

Whenever the archiving runs I either get a segfault or a zend_mm_heap when idsite = 2 is processed:


Archiving period = day for idsite = 2...
<?xml version="1.0" encoding="utf-8" ?>
<results>
        <result date="2011-01-29">255</result>
        <result date="2011-01-30">23</result>
        <result date="2011-01-31">8756</result>
        <result date="2011-02-01">9145</result>
.
.
.
        <result date="2011-03-18">6057</result>
        <result date="2011-03-19">267</result>
        <result date="2011-03-20">25</result>
        <result date="2011-03-21">6813</result>
</results>zend_mm_heap corrupted


.
.
.
        <result date="2011-03-18">6057</result>
        <result date="2011-03-19">267</result>
        <result date="2011-03-20">25</result>
        <result date="2011-03-21">6820</result>
</results>./archive.sh: line 73: 26943 Segmentation fault      $CMD

I’ve tracked this down to this one site by hard-wiring archive.sh to remove site 2 and then to just process site 2. All other sites are processed fine.

In Piwik 1.0.0 I never once received an error from archive.sh!

I’ve done some searching and #824 seems to match the problem. I’ve checked and this patch is definitely in 1.2.1 :frowning:

We were using PHP 5.3.2 so I upgraded to 5.3.6 but it’s still the same. We’re also using eAccelerator so I tried disabling that but the problem still exists.

PHP’s memory_limit is at the default 128MB - I’ve tried upping this to 256MB and 512MB without success.

I’d rather not have to recreate that site as there’s a lot of historic data that we require.

Does anyone have any ideas?

Many thanks in advance,

Neil.


(Peterbo) #2

Hi Neil,

is PHP running with suhosin patch? If so, please try to disable it for testing.

Peter


(Matthieu Aubry) #3

Usually when there is a segfault, we recommend to upgrade php and it fixes it.BUt it seems you are using the latest already… Peter’s suggestion might work?


(vipsoft) #4

Double check that eaccelerator is disabled. I don’t know anyone successfully running Piwik with eaccelerator.


#5

Hi,

Thanks for the quick responses.

Sadly not. It’s the pure, unadulterated source from php.net, downloaded yesterday.

I migrated Piwik to a new server in July last year and ever since it’s had eAccelerator installed, with no problems. I commented out the loading of the PHP module:


;extension=eaccelerator.so
;eaccelerator.cache_dir="/var/www/eaccelerator"
;eaccelerator.debug=0

and a ‘php -i’ shows that it’s definitely not loaded.

It seems that the segfaults have now ceased! The last one I received was at 00:02 - the archive run was scheduled in for 00:01. I guess that this is because it’s a new day and that there may have been something in yesterday’s data that it didn’t like? Would be nice to track this down, though.

Many thanks once again,

Neil.


(Thomas Seifert) #6

Well, there should be nothing in the data which could break php into pieces, aka leading to a segfault.
Could you try to get the archive run in a strace session and provide a backtrace out of it after it crashed?


(Peterbo) #7

I have been running Piwik on an eAccelerated Server ever since with a high load and many instances. Never had a problem with that.

@hillardn, remember that you have to recompile / rebuild eAccelerator when you update PHP! (make clean, make, make install), and that only eAccelerator 0.9.6.1 is supporting PHP 5.3+. On the other hand, there are also alternatives to eAcc.

Peter


#8

[quote=“Thomas Seifert”]
Well, there should be nothing in the data which could break php into pieces, aka leading to a segfault.
Could you try to get the archive run in a strace session and provide a backtrace out of it after it crashed?[/quote]

Will see what I can do - gdb isn’t currently installed on the server so I have to jump through hoops to get it installed :frowning:

Yeah, I missed recompiling it when I upgraded PHP but it whinged and the module wouldn’t load! I recompiled it and then it loaded fine. Definitely 0.9.6.1 as well:


This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies
    with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator

I’ll post again once I’ve got gdb installed. I’ve just run a quick strace on it (it’s now site 7 that’s failing) seems to indicate that the output is produced, followed by the segfault:


.
.
.
        <result date="2011-03-19">1679</result>
        <result date="2011-03-20">299</result>
        <result date="2011-03-21">22538</result>
        <result date="2011-03-22">8463</result>
</results>) = 2220
[pid 16592] close(6)                    = 0
[pid 16592] close(5)                    = 0
[pid 16592] poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
[pid 16592] write(4, "\1\0\0\0\1", 5)   = 5
[pid 16592] shutdown(4, 2 /* send and receive */) = 0
[pid 16592] close(4)                    = 0
[pid 16592] --- SIGSEGV (Segmentation fault) @ 0 (0) ---

Many thanks,

Neil.


#9

Hi,

Right, hoops jumped through and gdb now installed on the server! Here’s the backtrace, as requested:


.
.
.
        <result date="2011-03-25">21628</result>
        <result date="2011-03-26">1797</result>
        <result date="2011-03-27">276</result>
        <result date="2011-03-28">9301</result>
</results>
Program received signal SIGSEGV, Segmentation fault.
0x0834a78a in zend_mm_remove_from_free_list (heap=0x8879868,
    p=<value optimized out> ) at /usr/local/src/php-5.3.6/Zend/zend_alloc.c:830
830                             ZEND_MM_CHECK_TREE(mm_block);
(gdb) bt
#0  0x0834a78a in zend_mm_remove_from_free_list (heap=0x8879868,
    p=<value optimized out> ) at /usr/local/src/php-5.3.6/Zend/zend_alloc.c:830
#1  _zend_mm_free_int (heap=0x8879868, p=<value optimized out> )
    at /usr/local/src/php-5.3.6/Zend/zend_alloc.c:2034
#2  0x08372a6c in zend_hash_destroy (ht=0x8f15628)
    at /usr/local/src/php-5.3.6/Zend/zend_hash.c:534
#3  0x0835e0fa in destroy_zend_class (pce=0x9a5d0b4)
    at /usr/local/src/php-5.3.6/Zend/zend_opcode.c:187
#4  0x08372748 in zend_hash_apply_deleter (ht=0x8879b10, p=0x9a5d0a8)
    at /usr/local/src/php-5.3.6/Zend/zend_hash.c:614
#5  0x08372837 in zend_hash_reverse_apply (ht=0x8879b10,
    apply_func=0x8359d90 <clean_non_persistent_class> )
    at /usr/local/src/php-5.3.6/Zend/zend_hash.c:763
#6  0x0835ac78 in shutdown_executor ()
    at /usr/local/src/php-5.3.6/Zend/zend_execute_API.c:310
#7  0x08367153 in zend_deactivate ()
    at /usr/local/src/php-5.3.6/Zend/zend.c:890
#8  0x0831628f in php_request_shutdown (dummy=0x0)
    at /usr/local/src/php-5.3.6/main/main.c:1635
#9  0x083e6974 in main (argc=5, argv=0xbfffea44)
    at /usr/local/src/php-5.3.6/sapi/cli/php_cli.c:1374