Limitation on number of custom variable values / possible tuning variables?

Hello,

for segmentation purposes I’m trying to use custom variables.
I’ve got one variable where the value is between 1 and 30 million. Previously only 50 were shown, so I’ve raised
datatable_archiving_maximum_rows_referers to 30 million - as the inline docs mention that this value is used for the custom variables.
Now its showing more but still limited but there are some more variables like

datatable_archiving_maximum_rows_subtable_referers
datatable_archiving_maximum_rows_actions
datatable_archiving_maximum_rows_subtable_actions
datatable_archiving_maximum_rows_standard

Sure I could raise all of them but that makes archiving take ages and loooots of memory.
Which of those are really needed to get segmentation for all possible custom variable values as proposed above?

Thanks,

thomas

Technically you should only have to change:


datatable_archiving_maximum_rows_referers = 1000
datatable_archiving_maximum_rows_subtable_referers = 50

To higher values. If it’s not working as expected maybe there’s a bug. What is the limit you see?

It seems to be the limit from datatable_archiving_maximum_rows_subtable_referers .
I’ve now increased that to 30 million too, similar to datatable_archiving_maximum_rows_referers and now the archiving fails at the memory limit of 6GB with 2 million generated actions. I have to see how much memory it needs now to finish the archiving run :-(.

Ok, we can definitely improve this by applying this algorithm “in-memory update” to Custom variables: “Ranking Query” to decrease archiving memory consumption and transfered data between MySQL and PHP · Issue #3330 · matomo-org/matomo · GitHub

we can try work on this in few weeks & it would really improve memory usage a lot (for custom variables).