Data for week, month, year period not correctly archived/summed up

Premium Features
1

Hi everyone,

we are using the custom report plugin to report specific KPI to our internal stakeholders and customers.

After creating new reports in the beginning of december, I realized that the numbers for the weekly, monthly and yearly periods are not summed up correctly. I checked the numbers for each day and summed them up by hand, comparing the result to the reported weekly, monthly, yearly data archived by matomo. I found that it was not the same, even though it should be.

For the mentioned reports, I only used the metric “Hits”.

A few examples:

Report 1

  • Creation Date 1.12.2022
  • Weekly data from 28.11.-4.12: Data for the 28.11. and the 29.11. is missing
  • Monthly data for november 2022: Only the data of the 30.11. is used, the rest is missing
  • Yearly data 2022: Only data for the 30.11. and all days in dec (so far) are summed up, the rest (1.1. until 29.11.) is missing

Report 2

  • Creation date: 7.12.2022
  • Weekly data from 5.12.-11.12: Data for the 5.12. is missing
  • Monthly data for december 2022: Only the data until the 5.12. is used, the rest is missing
  • Yearly data 2022: Only data for 1.12. until 5.12., the rest (1.1. until 30.11. is missing)

Report 3

    1. Creation date: 14.01.2022
    1. Weekly data from 10.1.-16.1: Data missing for 10.-12.1.
    1. Monthly data for January: Data from 1.1.-12.1. missing
    1. Yearly data 2022: Data from 1.1.-12.1. missing

Here’s a little info from our configuration that might be related:

custom_reports_always_show_unique_visitors	0	
custom_reports_disabled_dimensions	''	
custom_reports_max_execution_time	0	
custom_reports_periods_force_aggregate_report_unique_metrics_evolution	''	
custom_reports_validate_report_content_all_websites	1	
datatable_archiving_maximum_rows_custom_reports	10000	
datatable_archiving_maximum_rows_subtable_custom_reports	5000
rearchive_reports_in_past_last_n_months	12

I’m not a programmer or a technical person, but from what I found it seems as if the archiving processes for week, month, year are not waiting for the day data to be fully processed. This results in not summing up all day data but only the one that’s archived by then?

I’d appreciate any help or ideas how to solve this!

Thanks :slight_smile:

2

Hi @janbir
I think you can get some answers there:

3

Hi Philippe,

thanks for answering that quickly :slight_smile:

I read this article before, but I was not aware that i always need to invalidate historical data in order to have proper custom reports? I was assuming that the custom reports plugin is working with already processed data from matomo core, since the data for the last 12 months (according to our setting rearchive_reports_in_past_last_n_months 12) is plausible and fine (for all periods), except the mentioned ones.

I would love to avoid having to (manually) invalidate data every time I set up a new report, thinking of our high traffic instance. Reprocessing data is a big overhead and raises the costs for the matomo infrastructure.

4

It is not possible for Matomo to preprocess all kind of data aggregation (it would be a very huge of aggregations). Think Matomo custom reports allow the mix of 3 dimensions on several tens of different dimensions, and the display of several metrics. The Cartesian product of all possible combination is really very huge…

5

Hi @janbir ,

Re-archiving will not happen automatically as it is necessary to invalidate to re-process historical data, it is possible to control which plugin reports you would like to process, also whether you would like to process specific segments by passing in the arguments in the command like * add –skip-all-segments to only process the “All Visits” segment

  • add - -force-idsegments if we only invalidated a specific segment above
  • add - -plugin where possible to only invalidate reports for a specific plugin. Eg. --plugin=“CustomReports” or --plugin=“Funnels”
1 Like