The important thing to understand, is that having any of the above is better than having no data to back a hypotheses; that’s when it can easily get into the realm of “hunch-based” hypotheses. This is something to steer away from, because we don’t want to invest our time and development resources into an assumption.
Moral of the story: some data is better than no data, because we are always working with a certain volume of numbers (either limited or in abundance), but the statistical rules applied to their significance are governed by man-made rules; so even if a hypothesis is backed by < 100 data points in a market research survey, that’s better than thinking “this might just work”.
Top Tips for Experimentation Analysts:
- Learn to explain things in a non-jargon fuelled manner. Stakeholders without a numerical/stats background will greatly appreciate this, so it is important to learn the art of translation!
- Understand that Optimizely isn’t a magic tool that should be used to validate our biases – because, if used that way, people will get fed up with not seeing the results they want. It is a tool to be used to constantly update our conceptions and ideas, in a directed manner.
- Explain uplifts with caution. 60-70% uplifts in conversion can seem appealing to stakeholders, but if it requires a time-consuming redesign of the site or product, is it going to be worth the development to permanently implement this? Watch out as they can be misleading!
If you have a hunch that changing something within a digital product may have a positive impact on performance metrics, perform analysis on the site and gather some data to rationalise your assumptions; this will help in understanding whether you are either utterly correct, or you are completely shooting in the dark.
Either way, both of these are useful manifestations because, as both analysts and human-beings in general, we don’t question how we’ve come to a certain conclusion – so it could be a useful exercise to simply validate or dismiss your assumptions.
Thank you to Frank Hopkins for sharing his inside knowledge with us, we really look forward to following your experimentation journey! If you are interested in a role like this, please see our Analytics Jobs and check out our North West Marketing Analytics Sector Skills Survey!