I have created an A/B test and have been running the test for a few days, but have found the report produced difficult to interpret. I have tried reading the documentation, but am none the wiser. Here is what I understand the report to be telling me, but would appreciate validation that I have understood the report correctly.
As background, we are using matomo to track a site. I have created an AB test and added the script to some of the pages on the site, not all of which meet the target pages filter.
Here is what I understand the report to be telling me.
Evolution over the period: This is showing the number of pages that included the AB testing script. The script is only included on some of the pages on the site, but not all of them. This is different to the number of pages included in the test as the script is included on pages that are not part of the target pages.
In the overview report I have the following columns:
Visits - this is showing the number of visits to pages that included the AB testing script, not the number of visits to the site in general.
Visit actively entered - this is showing the number of pages that met the conditions of the target pages filter.
Unique Visitors - this is showing the number of unique visitors that visited pages that met the conditions of the target pages filter.
Of the success metrics they have the following meaning.
Average Pageviews: This is the number of pages viewed across the whole site and not just those that have the AB testing script included.
Bounce Rate: this is the percentage of users that viewed the page meeting the target pages filter and then did not view any more pages on the site.
Average time on website: This is the average time the user spent on the whole site and not just the time spent on the pages with the AB testing script installed.
The pageviews, visits, bounce rate match pretty much simply the same metrics as in Matomo https://glossary.matomo.org. So Bounce rate does not check if they viewed any other pages besides target pages. And average time on website is indeed spent time on the whole site.
Visits entered: This is the number of visits that viewed one of your matching pages and took part in the A/B test.
Generally the most important metrics are
Detect effect → this percentage shows you if a variation performs better or worse and by how much
Statistical significance → you want this typically > 95% otherwise the detected effect might not be due to this change
Remaining visitors → you want to wait until this is zero or a dash basically
And you also want to make sure to run the test long enough depending what you want to test and how much traffic / conversions you have. Not just a few hours as eg during the day your visitors might behave completely different than at night. Or if you run it for multiple days then visitors might behave differently during weekends and weekdays.
I suggest you use some goals in the success metrics but this depends really on your business/website.
I am still confused about the visits column shown in your report. We get around 50,000 visits to the website a day, but my AB test is recording visits as just over 1000 for the whole period of the test (several days). Why the difference if they are the same figure. Here is one of the conversion goals from my report
The number of visits is indeed not the overall visits in your Matomo. So those are the number of visits that took part in your experiment. It’s higher than the unique visitors since one visitor can visit your site multiple times. The difference to “Visits entered” is that they might not always view one of the pages that are part of your experiment.
