Reloading data in from the web logs

Bluebird63 · March 17, 2020, 10:20pm

Hi guys,

It looks I’ve had disk space issues on my Matomo DB server which has stopped a fair amount of data being written to the db since 6 am this morning

The server is all sorted now but we have a gap in the analytics. Having said that as we have a separate web-server, we have all of the raw logs for all of the analytics. Is there a way that I can reload the database from these logs?

Many Thanks

Lukas · March 18, 2020, 9:39am

In case you have the access.log of you Matomo instance in that time range (not the site that was tracked), you can use this really obscure feature to send all this data again to Matomo:
https://matomo.org/faq/log-analytics-tool/faq_19221/

Bluebird63 · March 18, 2020, 10:23am

Hi Lukas, many thanks for the response

I have the logs in place and executed the python script but hit a snag - I received this error;

“Fatal error: the selected log format doesn’t include the hostname: you must specify the Matomo site ID with the --idsite argument”

Issue is, I’ve got tracking stats for a few different websites in this log file (mapped to different site IDs) so if i add the --idsite argument, will it move all of the data into that specified site ID? Obviously i want the data flowing to their correct Site Ids in the database.

Many Thanks in advance!

Lukas · March 19, 2020, 8:48am

Hi,

According to the code:

github.com

matomo-org/matomo-log-analytics/blob/953277eb27d8c68c363f1fdd09fc031d2b45ff33/import_logs.py#L1741-L1748


def check_format(self, format):
    if config.options.replay_tracking:
        pass
    elif format.regex is not None and 'host' not in format.regex.groupindex and not config.options.log_hostname:
        fatal_error(
            "the selected log format doesn't include the hostname: you must "
            "specify the Matomo site ID with the --idsite argument"
        )

you should never get to that error message if you added the --replay-tracking parameter.

Bluebird63 · March 26, 2020, 7:20pm

Thanks Lucas, as ever to the rescue!