A/B Test Randomization Not working

Hi,

I’m currently running an A/B test for two different versions of our website’s homepage. Each visitor to the website has a 50% chance of seeing either version. This randomness is set up with the A/B testing plugin.

At the moment, it seems like the A/B testing plugin is consistently favoring one version over the other. Right now, 29% of visitors are being directed to the original homepage, while 71% are seeing the second version. This is far from desirable.

This isn’t the first time this has happened; the same pattern occurred in a previous test.

Could you please help me understand:

  • How the plugin decides which version to show to visitors?
  • What might be causing this uneven distribution?
  • Is there anything we can do to fix this?

Hi @tharry ,

A/B test redirects to random variations. Difference will reduce when the test is nearing its end and it will even out once the test is ended as per your configuration. You may see a difference when the test is in progress and it is an expected behaviour.

Hi @karthik ,

The A/B test finished an these are the results.

The number of visits didn’t even out in the end.

I tried one of the changes suggested by one of the Matomo support staff:

And this was the result:

Basically, every visitor to the website starts with the original version, and it’s redirecting correctly. I can see this in the visitor log. However, there’s no record of people entering the second homepage in the A/B testing tool because I set a rule that excludes any visitor who enters the “newhomepage” site from being included in the experiment.

@karthik

As a PhD student, I’m running a test on this tool before my main research begins. I’ve spent over a month figuring out how it works. Can I get help from someone on your team via a video call to set up my experiment correctly and move forward on this?

Hello @tharry Can you please reach out to shop@matomo.org so we can investigate this further? Please copy the link of this forum for reference.

Thank you. @Emerson_N I have just done that.