Segfault when running archive.sh :(

Support & Bugs
1

Hi,

Today I upgraded our installation of Piwik from 1.0.0 to 1.2.1. Apart from it taking a long time to upgrade the (4.8GB) database it went smoothly. Until I re-enabled the archiving…

Whenever the archiving runs I either get a segfault or a zend_mm_heap when idsite = 2 is processed:


Archiving period = day for idsite = 2...
<?xml version="1.0" encoding="utf-8" ?>
<results>
        <result date="2011-01-29">255</result>
        <result date="2011-01-30">23</result>
        <result date="2011-01-31">8756</result>
        <result date="2011-02-01">9145</result>
.
.
.
        <result date="2011-03-18">6057</result>
        <result date="2011-03-19">267</result>
        <result date="2011-03-20">25</result>
        <result date="2011-03-21">6813</result>
</results>zend_mm_heap corrupted



.
.
.
        <result date="2011-03-18">6057</result>
        <result date="2011-03-19">267</result>
        <result date="2011-03-20">25</result>
        <result date="2011-03-21">6820</result>
</results>./archive.sh: line 73: 26943 Segmentation fault      $CMD

I’ve tracked this down to this one site by hard-wiring archive.sh to remove site 2 and then to just process site 2. All other sites are processed fine.

In Piwik 1.0.0 I never once received an error from archive.sh!

I’ve done some searching and #824 seems to match the problem. I’ve checked and this patch is definitely in 1.2.1 :frowning:

We were using PHP 5.3.2 so I upgraded to 5.3.6 but it’s still the same. We’re also using eAccelerator so I tried disabling that but the problem still exists.

PHP’s memory_limit is at the default 128MB - I’ve tried upping this to 256MB and 512MB without success.

I’d rather not have to recreate that site as there’s a lot of historic data that we require.

Does anyone have any ideas?

Many thanks in advance,

Neil.

2

Hi Neil,

is PHP running with suhosin patch? If so, please try to disable it for testing.

Peter

3

Usually when there is a segfault, we recommend to upgrade php and it fixes it.BUt it seems you are using the latest already… Peter’s suggestion might work?

4

Double check that eaccelerator is disabled. I don’t know anyone successfully running Piwik with eaccelerator.

5

Hi,

Thanks for the quick responses.

Sadly not. It’s the pure, unadulterated source from php.net, downloaded yesterday.

I migrated Piwik to a new server in July last year and ever since it’s had eAccelerator installed, with no problems. I commented out the loading of the PHP module:


;extension=eaccelerator.so
;eaccelerator.cache_dir="/var/www/eaccelerator"
;eaccelerator.debug=0

and a ‘php -i’ shows that it’s definitely not loaded.

It seems that the segfaults have now ceased! The last one I received was at 00:02 - the archive run was scheduled in for 00:01. I guess that this is because it’s a new day and that there may have been something in yesterday’s data that it didn’t like? Would be nice to track this down, though.

Many thanks once again,

Neil.

6

Well, there should be nothing in the data which could break php into pieces, aka leading to a segfault.
Could you try to get the archive run in a strace session and provide a backtrace out of it after it crashed?

7

I have been running Piwik on an eAccelerated Server ever since with a high load and many instances. Never had a problem with that.

@hillardn, remember that you have to recompile / rebuild eAccelerator when you update PHP! (make clean, make, make install), and that only eAccelerator 0.9.6.1 is supporting PHP 5.3+. On the other hand, there are also alternatives to eAcc.

Peter

8

[quote=“Thomas Seifert”]
Well, there should be nothing in the data which could break php into pieces, aka leading to a segfault.
Could you try to get the archive run in a strace session and provide a backtrace out of it after it crashed?[/quote]

Will see what I can do - gdb isn’t currently installed on the server so I have to jump through hoops to get it installed :frowning:

Yeah, I missed recompiling it when I upgraded PHP but it whinged and the module wouldn’t load! I recompiled it and then it loaded fine. Definitely 0.9.6.1 as well:


This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies
    with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator

I’ll post again once I’ve got gdb installed. I’ve just run a quick strace on it (it’s now site 7 that’s failing) seems to indicate that the output is produced, followed by the segfault:


.
.
.
        <result date="2011-03-19">1679</result>
        <result date="2011-03-20">299</result>
        <result date="2011-03-21">22538</result>
        <result date="2011-03-22">8463</result>
</results>) = 2220
[pid 16592] close(6)                    = 0
[pid 16592] close(5)                    = 0
[pid 16592] poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
[pid 16592] write(4, "\1\0\0\0\1", 5)   = 5
[pid 16592] shutdown(4, 2 /* send and receive */) = 0
[pid 16592] close(4)                    = 0
[pid 16592] --- SIGSEGV (Segmentation fault) @ 0 (0) ---

Many thanks,

Neil.

9

Hi,

Right, hoops jumped through and gdb now installed on the server! Here’s the backtrace, as requested:


.
.
.
        <result date="2011-03-25">21628</result>
        <result date="2011-03-26">1797</result>
        <result date="2011-03-27">276</result>
        <result date="2011-03-28">9301</result>
</results>
Program received signal SIGSEGV, Segmentation fault.
0x0834a78a in zend_mm_remove_from_free_list (heap=0x8879868,
    p=<value optimized out> ) at /usr/local/src/php-5.3.6/Zend/zend_alloc.c:830
830                             ZEND_MM_CHECK_TREE(mm_block);
(gdb) bt
#0  0x0834a78a in zend_mm_remove_from_free_list (heap=0x8879868,
    p=<value optimized out> ) at /usr/local/src/php-5.3.6/Zend/zend_alloc.c:830
#1  _zend_mm_free_int (heap=0x8879868, p=<value optimized out> )
    at /usr/local/src/php-5.3.6/Zend/zend_alloc.c:2034
#2  0x08372a6c in zend_hash_destroy (ht=0x8f15628)
    at /usr/local/src/php-5.3.6/Zend/zend_hash.c:534
#3  0x0835e0fa in destroy_zend_class (pce=0x9a5d0b4)
    at /usr/local/src/php-5.3.6/Zend/zend_opcode.c:187
#4  0x08372748 in zend_hash_apply_deleter (ht=0x8879b10, p=0x9a5d0a8)
    at /usr/local/src/php-5.3.6/Zend/zend_hash.c:614
#5  0x08372837 in zend_hash_reverse_apply (ht=0x8879b10,
    apply_func=0x8359d90 <clean_non_persistent_class> )
    at /usr/local/src/php-5.3.6/Zend/zend_hash.c:763
#6  0x0835ac78 in shutdown_executor ()
    at /usr/local/src/php-5.3.6/Zend/zend_execute_API.c:310
#7  0x08367153 in zend_deactivate ()
    at /usr/local/src/php-5.3.6/Zend/zend.c:890
#8  0x0831628f in php_request_shutdown (dummy=0x0)
    at /usr/local/src/php-5.3.6/main/main.c:1635
#9  0x083e6974 in main (argc=5, argv=0xbfffea44)
    at /usr/local/src/php-5.3.6/sapi/cli/php_cli.c:1374