Sample Size and Significance

I recently conducted a test for an electronics retailer whose focus is on selling protection plans and accessories for computer products like iPads and laptops. Having worked for a major electronics manufacturing company doing analytics I know the profit margins on hardware (save Apple products) is astonishingly thin. Like paper thin. Like grocery store thin. The fat margins are on the accessories and protection plans. So we did a common sense test and made the area on product detail pages for laptop and tablet accessories and protection plans more prominent.

Now, the take rate for these products—particularly the critical protection plans—is very low. The test, if run according to the rules I learned from my stats professor (Art Vittorito you were my favorite professor!), should have a sample of at least 100 actions on the KPI per creative treatment or user experience. That means this test would have run for months on end to reach that goal.

This was a multivariate test. So you look at the results on the variable level as opposed to the creative level. The results for one variable were so off the charts positive (over 100% lift in desired user action) that even though the sample was lower than desired, you just have to run with it. Yes, it is possible to flip a coin and have it come up heads 20 times in a row. But 40 times in a row? 60 times? And the contributing factor was that it just made sense. The KPI content was part of a series of tabs. We brought that tab to the far left where the natural eye scanning starts (we read left to right of course). So the insane positive lift numbers + duh factor = calling the test before good statistical modelling sample is in.

Sometimes you don’t have to wait.