Experimentation Analysis with Frank Hopkins

My name is Frank Hopkins and I am an Experimentation Analyst for the BBC. For the last 5 years, I’ve worked as an “experimenter” of some capacity; with a background in exercise mechanics and statistics.

My role focuses on best experimentation practice across all BBC products – ranging from iPlayer to CBBC and Newsround. This often requires me to get in and amongst teams and help in the experimentation process, and on any given day I could be helping with hypotheses building, prioritisation, QA’ing experiments or performing post-hoc analysis to harness meaningful insight from experimentation data. 

Due to the size of the BBC, work is both genuinely and generally exciting on a day-to-day basis. As part of the BBC, I have worked with a variety of experiments, which have differed in size and complexity; from large-scale MVT experiments and homepage re-designs down to smaller and more nuanced changes on sites, such as re-ordering modules and prioritizing certain content items.

For those of you who work in digital, you may be familiar with the term “experimentation” or may have heard it being thrown around. If not, I will briefly define experimentation and what people mean (or think they mean!) when they say “let’s test this”.

"Experimentation, in a digital context, refers to the execution of a controlled, balanced experiment on a page, site and/or app where equal distribution of traffic is delivered to two or more variations of the same entity."

The purpose of experimentation is often misconstrued and it’s important to stress where its true value lies. To help “product” people understand the true value of experimentation and how it should be implemented, I take myself back to my days as an applied scientist, studying Biomechanics.

In an applied science context, the body of literature is examined, a gap or unanswered questions are identified, a reasonable hypothesis is presented, appropriate methods for answering the hypothesis are conducted and statistical analysis is performed. Following the results, conclusions are drawn and future research is proposed. In theory, this is how experimentation should be conducted in a digital and/or business setting. Experimentation tools should be used to test scientifically or numerically backed hypotheses. Such backing takes many forms, including holistic analysis performed by central analysts and/or data scientists, field/market research, qualitative analysis or UX analysis.