Deleted Browser Plugins data

Support & Bugs
1

How is it possible to delete the Browser Plugins data in the database? Maybe with a MySQL command on the command line? Or via a PHP script.

2

Hi @melbao

Which plugin is it exactly?

Did you:

  1. Disable the plugin?
  2. Delete the plugin form the Matomo installation?
  3. Tried to apply the $ ./path/to/matomo/console core:update command (see link below)?

See:

If this doesn’t work, I think you’ll have to remove additional tables / fields by hand with MySQL command…

3

Its a little bit mistake? I mean the data was tracked about the in a visitor browser installed plugins like: cookie, JavaScript, Flash, …

I had reorganized my tracking. Before, Browser Plugin data was tracked. Yet, Browser Plugin data was not tracked. Yet i will delete this data afterwards in the saved data. Because, i have never used this and will not used this. Is this data saved in a single database table or in the visit log table?

4

Hi @melbao
Looking at the log_visit table description, plugin data are stored in following fields:

  • config_resolution: a string identifying the screen resolution the visitor used to make this visit (eg, '1024x768')
  • config_pdf: whether the visitor’s browser can view PDF files or not
  • config_flash: whether the visitor’s browser can view flash files or not
  • config_java: whether the visitor’s browser can run Java or not
  • config_director:
  • config_quicktime: whether the visitor’s browser uses quicktime to play media files or not
  • config_realplayer: whether the visitor’s browser can play realplayer media files or not
  • config_windowsmedia: whether the visitor’s browser uses windows media player to play media files
  • config_gears:
  • config_silverlight: whether the visitor’s browser can run silverlight programs or not
  • config_cookie: whether the visitor’s browser has cookies enabled or not

Removing these tracked data won’t probably save much space (as columns won’t disappear, I think).
Did you disable the DevicePlugins plugin?

5

i have lost the overview…

You mean the DevicePlugins. OK. I have found it in the settings. It was Activated.

Last year, i expand the tracking code to:

_paq.push(['disableCookies']);
_paq.push(['disableBrowserFeatureDetection']);

Yesterday, after multiple manually core:archive and after multiple manually core:invalidate-report-data the database was shrunk to less 300 MB. Under considering, that invalidate is included in archive?

Hint: a multiple core:archive is neccessary, because by one not all data is really archived. By me, round about 5 core:archive runnings were neccessary. Yet, a hourly core:archive log is only ~15 Kb. (before 2-1 MB).

So, i observed old data. Yet no Browser Plugin data is visible. How can this be explained? I don’t have changed the settings or plugins.

After that, (before a fresh database backup) i have deactivated the DevicePlugins, and run at multiple times a core:archive, and make a new database backup: no difference in the file size.

Curiously, i don’t find a separated statistic over the Browser Plugin data. Does such a one exist?

I don’t know what’s going on now. Was the Browser Plugins data deleted or not?

6

… its a ungratefully job …

so, i have started the “Anonymize previously tracked raw data” in the matomo backend settings, Privacy, Anonymize data. There are a little bit mistakes. In my matomo installation (4.14.2) there is no config_director and no config_gears in the dropdown menu. The config_resolution may referenced with DevicePlugins, but is used in the visit logs by “Device (type, brand, model)”, in the black popup. The resolution is also not really a device plugin. Further, the resolution is no longer saved with _paq.push(['disableBrowserFeatureDetection']);. And the resolution is not saved as an integer in the reports, as the other mentioned info. Its a little bit chaos in matomo?

so … with this action, only an integer is changed. So the effort is not really worth it. The action is too time expensive. Then, after that, the/all reports must be invalidate and after that the/all reports must be re-processed. Very time expensive.

… that was a short interim message. The matter is still ongoing and is far from over.

What is needed is a command line command for the “Anonymize previously tracked raw data”, because the info “The anonymization should typically start within one hour.” it is without that wait time expensive enough.

7

… yet i have a big problem, the “Previous raw data anonymizations” don’t start …

8

The console command for manually start:

privacymanager:anonymize-some-raw-data [--date="..."] [--unset-visit-columns="..."] [--unset-link-visit-action-columns="..."] [--anonymize-ip] [--anonymize-location] [--anonymize-userid] [--idsites="..."]

not required, default all:
–date
–idsites

comma separated.
in quotes only when withespaces exists
–unset-visit-columns
–unset-link-visit-action-columns

used blank without value:
–anonymize-ip
–anonymize-location
–anonymize-userid

example for --unset-visit-columns:

php /path/to/matomo/console privacymanager:anonymize-some-raw-data --unset-visit-columns=config_cookie,config_flash,config_java,config_pdf,config_quicktime,config_realplayer,config_silverlight,config_windowsmedia,config_resolution > /path/to/mylogs/unset-visit-columns-$(date +"%Y%m%d%H%M%S").log

Dond’t work, because:

Start date is “1996-01-01 00:00:00”, end date is “2023-04-21 23:59:59”
e[30;46mAre you sure you want to unset the log_visit columns “config_cookie, config_flash, config_java, config_pdf, config_quicktime, config_realplayer, config_silverlight, config_windowsmedia, config_resolution” for all visits between “1996-01-01 00:00:00” to “2023-04-21 23:59:59”? This action cannot be undone. Type “OK” to confirm this section.e[39;49m

there is no OK button in the console … boring.

Fix:
…
matomo/plugins/PrivacyManager/Commands/AnonymizeRawData.php

    private function confirmAnonymize(InputInterface $input, OutputInterface $output, $startDate, $endDate, $action)
    {
        $noInteraction = $input->getOption('no-interaction');
        if ($noInteraction) {
            return true;
        }
        // $dialog = $this->getHelperSet()->get('dialog');
        // $value = $dialog->ask(
            // $output,
            // sprintf('<question>Are you sure you want to %s for all visits between "%s" to "%s"? This action cannot be undone. Type "OK" to confirm this section.</question>', $action, $startDate, $endDate),
            // false
        // );
        // if ($value !== 'OK') {
            // return false;
        // }
        return true;
    }

The command runs very fast in under 1 second?

Start date is "1996-01-01 00:00:00", end date is "2023-04-21 23:59:59"
Running behaviour on all sites.
Applying this to visits between '1996-01-01 00:00:00' and '2023-04-21 23:59:59'.
Starting to unset log_visit table entries.
Number of unset log_visit table entries: 123456789
Starting to unset log_conversion table entries (if possible).
Number of unset log_conversion table entries: 0
Done

But… this looks like the datas are not deleted in the raw data, but only in the reports? Why is the name then anonymize-some-raw-data?

example for --anonymize-ip:

php /path/to/matomo/console privacymanager:anonymize-some-raw-data --anonymize-ip > /path/to/mylogs/anonymize-ip-$(date +"%Y%m%d%H%M%S").log

This command works longer (~10 minutes, depending on the amount of data) with permanent CPU usage:

Start date is "1996-01-01 00:00:00", end date is "2023-04-22 23:59:59"
Running behaviour on all sites.
Applying this to visits between '1996-01-01 00:00:00' and '2023-04-22 23:59:59'.
Starting to anonymize visit information.
Number of anonymized IP and/or location and/or User ID: 123456789
Done

After this actions the reports must be invalidate:
CONSIDER: set the second date as the current date!

php /path/to/matomo/console core:invalidate-report-data --dates=1996-01-01,2023-04-22 --sites=all > /path/to/logs/matomo-invalidate-report-data-$(date +"%Y%m%d%H%M%S").log

This runs time expansive (~30 minutes, depending on the amount of data) with permanent CPU usage.

Last but not least the reports must be generated new with core:archive.
BEFORE it is better to deactivate the core:archiv triggering by browser, because otherwise the jobs are run to often at the same time.
Disable browser triggers for Matomo archiving

WARNING: The core:archiv jobs are RAM expansive. When its possible, then set the php memory_limit at the greatest value. 128MB is to small, better 256MB or 512MB.

php /path/to/matomo/console core:archive --force-all-websites --url=https://mtm.jiandu.de/ > /path/to/logs/matomo-archive-$(date +"%Y%m%d%H%M%S").log

Or run this as a cronjob / crontab:
In the example every hour at minute 20.

20 * * * * php /path/to/matomo/console core:archive --url=https://mtm.jiandu.de/ > /path/to/logs/matomo-archive-$(date +"\%Y\%m\%d\%H\%M\%S").log

The core:archive does not do everything at once, but in several runs. After a few times then everything is done and the jobs do only the latest data. The first logs are big, the further smaller and smaller. The first runs are very time expensive (~1-2 hours, depending on the amount of data), the last only few seconds ore minutes.
When the log files are every time small (few kb), it is possible to deactivate the cronjob and activate the core:archiv triggering by browser, or use both.

no guarantee, no warranty.

Edit: by problems inside the log files use --no-ansi in the commands.