While this method sounds really easy to conduct, actually there are many things that can go wrong during an A/B Testing process. To run a proper A/B Test, we need for example:
- hypothesis to test
- big enough sample size
- to run the test for full weeks
- a version identical with the original in order to see the noise (noise here means that if you have two identical UI solutions, and you split the users into two groups, you will still get some differences because of random factors).