CRAP Talks 20 - Connor Wilkinson: Fireside Chat on Experimentation
- Friday, 12 June 2026
- London
Speakers
Not in your language? Your browser can translate this page — use its built-in translate option (the translate icon in the address bar, or the right-click menu).
Summary
Parveen Downer interviews Connor Wilkinson, Head of eCommerce Optimization at Asda, on experimentation and A/B testing. Connor discusses Asda's ecommerce journey, organizational structure that spans both online and physical retail operations, and examples of experiments ranging from picker optimization to substitute product algorithms. The conversation explores how to maintain experimental rigor through data-driven hypotheses, build experimentation culture across large organizations, and navigate the unique challenges of testing in physical retail environments.
Key quotes
“We want to be the best hypothesizers in the business.”
Connor Wilkinson · 8:12 “A bad hypothesis is a guess, basically.”
Connor Wilkinson · 9:06 “I don't think we should democratize how we analyze those tests.”
Connor Wilkinson · 11:36 “CRO is skill. Experimentation is a mindset.”
Connor Wilkinson · 23:23
Chapters
Transcript
View as markdownIntroduction and Asda's Ecommerce Journey
All right, thank you. Thank you.
You haven't seen the content yet. Okay, so hi everyone, my name is Parveen. I am a senior product manager in the health tech space, a startup here in London. And I've been an organizer of crap Talks for over four years now.
2017 I started and I'm also a product management interview coach. So this is me making myself up because I don't do that enough.
Yeah, there. Thank you.
So my background's actually in healthcare and education operations and I moved into tech about four years ago. Four or five years ago.
And obviously really loving this space and very pleased to have the opportunity to chat with Connor today. So Connor is the head of E commerce optimization at Asda. Yes, that Asda, big, big ups. He is in charge of the team that oversees all things experimentation on the grocery side of the business.
So you probably know Asda as being having other brands as well, like George the clothing brand for example. But he is all things groceries and interestingly he has a unique remit of not only in person online experiences, but some of his remit touch's in person experiences which we will touch on later today. So we're very fortunate to have Connor here today. Can you please give him a big warm round of applause?
All right, so we're going to dive in because we don't have a huge amount of time, but also let's just get in situation. So given that Asda is a huge retailer in the brick and mortar space, what has been asda's journey into E commerce?
Cool. So hi everyone and thanks for having me.
First thing is I'm from Northern Ireland, so if anyone can't understand me, just put your hand up and I'll say it in English you can understand.
So yeah, so as the believe it or not, we're second biggest online retailer in the grocery space in the uk.
We're behind Tesco. Tesco, they won the space race. So they've just got so many more stores than us. But we are pretty much, we've been second in the market now for a long time.
And believe it or not, in 1999 we released a CD Rom with a list of products on which you could call our call center and place an order.
And then the following year we launched our first website and we started taking grocery orders back in 2000. And so I know we're more than 3 billion pound business now just in online grocery and we're still only 12% of the total business. So gives you sort of an idea of how big ASDA are. As a retailer, Yeah.
I think we were very fortunate to have a speaker was it last year from a different online retailer who claimed to be the first E commerce website. Does anyone remember or do you know who this business is?
The Vatican? No, no. Pizza Hut. Yeah, Pizza Hut.
Anyways, just fun fact from a throwback.
Okay, so for the first time at crop talks we have the opportunity to get some unique insight on experimentation in the physical space.
Organizational Structure for Experimentation
So I'd love to know how have you set up your organizational structure to enable experimentation in both the E commerce and physical space?
Yeah. Cool. So when most people talk about E commerce they think about a website. Now that is true.
Obviously we take orders through the website, but the business in ASDA is set up as an E commerce business.
Everything from the store operations. So how much space we have in store, how many totes we put into store, where the stores are right the way through to the website, creative monetization, et cetera, et cetera.
So my team essentially sits across that full horizontal and then we've got vertical teams across each of those. So last mile we've got operating model, etc. And my team sit right the way across those to help with supplying data, supplying analytics, helping with trying new tools and new things out. And quite a lot of those things are in store because that's where the money is.
If you save, if you save a pound or you save a pick point of how quickly you pick an item that's worth an awful lot more than selling another item on the front end of the website.
So the operating model is a big part of the organization. And my team actually built the system that routes a picker around shop floor.
And that's all built with Scala in actual fact.
And the guys built that model to operate that send the pickers around the shop floor. And we do an awful lot of experimentation to make sure those pickers are doing the right, making the right the way around the shop floor, making sure the maps are correct. So we do a lot of testing there and what we do is we have 430 stores that we pick out of. And obviously when you've got that many stores, there's quite a lot of stores that look the same in terms of size, geography, customer customer loyalty, etc.
And we use data to make sure that we've got right control groups to send a new product into and we'll launch a product in those stores. So for example, we had a new pick trolley that put an extra couple of totes on the pictroly to see whether that Increased pick speed. Of course it did because you can pick more stuff. But what we didn't know is that added weight to too trolley and pickers thought that was too heavy and it was quite close to health and safety guidelines, etc.
Etc. So things like that. So we do a lot of that. That was a long winded answer.
Examples of Experimentation and Testing
But no, in the end you answered my next question, which is what are some examples of things you experiment with? Do you have a couple of other examples of things you've experiment with?
Yeah. So the picture example or anything to do with that kind of data, we predict quite a lot of it. So we use regression, we use a lot of data science modeling to work out what would happen if something happened. That gives us a lot more guidance on how we actually approach these products.
But another example would be we also have.
Does anyone do online grocery shopping?
Does anyone get a substitute? Yes, is a real pain point for people. My team built the subs engine in asda. So we find lookalike products and we make sure that those are acceptable.
We put them into a tree for the pickers to pick. That saves a lot of time rather than them sitting on the shelf edge and going, ooh, what should I pick instead of this one? And we can change those trees based on a lot of different algorithms and different rules. So that's another example.
Is return rate one of the things you look at?
It might be, yes.
Okay, very cool.
Maintaining Experimental Rigor and Culture
So it's all good and well to have obviously your experiments and everything set up, but an experiment is only as good as how well it's been planned and thought out. So I'd love to know, how do you keep people honest and reducing biased when creating experiments?
Yeah, it's a great question. Now, obviously with working in such a competitive market, it's very difficult to hold change back. Now, you don't really want to hold change back, but you don't want to do the wrong thing.
So the answer to that is very, very difficult. It's very, very difficult.
What we're doing right now is the structure that I was talking about there is actually pretty new. So we're going through that process right now and trying to embed that culture of experimentation. And what we try and do is, and what I give my guys is a bit of a North Star, if you will, is to try. We want to be the best hypothesizers in the business.
And if you got the right hypothesis, you set yourself up really well to understand what you're trying to do. And hypothesis is driven from data. It's not just a guess.
So we try and instill, you know, we sort of say, you know, come up with a hypothesis. We try and help them with a hypothesis. Kit, Craig Sullivan's hypothesis Kit, is a great one. So.
And we try and instill that into the guys. And if they. And if people come back to us with a hypothesis that is more of a.
We think it's going to do this. It's like, well, where's your data?
And try and help them to find that data Again, I don't know if that answers the question.
Yeah, yeah. Actually, you're sort of going on to the next impromptu question.
What is a bad hypothesis? Can you give us an example and like, pick it apart?
Yeah, well, a bad hypothesis is a guess, basically. So if you come up with an idea, something's going to happen and you have no idea whether it's going to happen or not. That's a bad hypothesis. But that is what people think a hypothesis is, believe it or not.
Well, everyone knows, everyone here can see that happening probably day in, day out. But that's a bad hypothesis when you don't have any data to back it up. Now, there is room for that because obviously you want to try stuff out, you want to see whether that guess works. But it's an initial idea of how to get to a hypothesis rather than a hypothesis.
Sort of pre work on the hypothesis. Yeah, necessary.
Okay.
So there's a lot of people who don't work in experimentation at asda. It's a very big company. So I wonder how do you influence or get people to become believers in experimentation given your sort of structure in the business?
Yeah, it's probably the biggest challenge we have, partly because if you're not.
People feel like if they're not doing something, they're not moving, they're not doing anything.
So that doesn't make sense. But.
But if they're not trying something out, they're not failing. They're not trying. And we obviously have a lot of monetary goals and that kind of gets in the way of actually sitting going. Has this actually worked?
So the short answer is we haven't solved that yet. We're still working through that. But what I'm trying to do is get pockets of teams who maybe haven't thought about hypothesizing, haven't thought about trying something and doing it in a controlled way. And I've got a nice little pocket across those verticals I was talking about before that are really starting to champion.
Yeah, this actually works.
This is helping us to make decisions.
Is there a certain philosophy you have around experimentation and culture?
Yeah, so the.
I do this is very topical. So obviously democratization of testing is. Or experimentation is quite hot topic at the minute. I completely agree that we should democratize testing.
However, I don't think we should democratize how we analyze those tests. I think that's still. Is a skill set that needs to be part of that horizontal team. So what I'm trying to instill is, you know, the guys who are making the changes, the guys who are trying to build different products, don't change what you're doing, just change how you're doing it.
That seems to be working certainly with those little pockets that I'm picking up.
Yeah, they've seen a lot of nodding heads here, especially with your analytics comment. We'll keep that in house.
Yeah, yeah, indeed.
So you have a very large team, of course, as does a big company. You mentioned about the different verticals and how your team is more horizontal. So how do you differentiate the types of roles on your teams? You have analysts, I think you mentioned you have data scientists as well.
How do you differentiate what their roles and responsibilities are?
Yeah, well, believe it or not, my team's not. My team isn't that big, but my, my team is set up to essentially I've got a part of the team that's responsible for operational type optimization at the minute. They're focusing mainly on MIBI and reporting just because we're in a bit of a transition with moving away from Walmart. So that team will be responsible for helping the operation side of the business.
So where we put operations, how we pick, where we put vans, how many vans we put in different stores and they're going to be focused mainly on that. And then the other side of the team will focus mainly on the front end, helping with, you know, different creative, different aspects of conversion, different aspects of usability of the site and helping with the product function as well. So we have a product function and then central to that, and I call them the engine room, I have a data science function.
And those guys are pretty much, you know, they're the clever guys that I could never do their job. So they're the guys who are, you know, predicting what's going to happen, making sure the data is in the right place, working with the central data teams to make sure that we've got the right tools, the right functionality and really driving any of those optimizations forward.
And is there a particular team that's responsible for keeping the data clean? And, and collecting clean data.
Yeah, I mean, we're kind of, don't use this phrase lightly. We're kind of marking our own homework at the minute because there isn't anyone else to do it. Believe it or not, we're in the process of transitioning from having all that served to us by Walmart, and we're bringing that in house. So there's a central data team in ASDA who will do that, but we're kind of in a bit of a transition at the minute.
So again, that's where that operations function will sort of migrate that across into a more central function.
Right. Okay. So we're going to change tack a little bit and let's get into the weeds a bit.
The Substitute Engine Case Study
So what is an experiment that you've run or your team has run that produced the most surprising results and did you change anything in the business because of it?
Yeah. So the one that springs to mind. I'll go back to the subs engine. It's a very topical internal discussion at the minute, but the algorithms we use are very wisdom of the crowd.
So it takes a lot of the data, builds the trees based on most of the customers who get substitute and accept the substitute. What we did is we added in some personalization algorithms in there, so things like brand preference, how often you would buy the same product, things like that.
And we put that, we tried that in the way that I described before.
And what it actually showed us was it worked. So customers were rejecting them far less. However, pickers were getting in the way of actually allowing the. The sub to happen.
So they were seeing, you've got to remember, they're picking the same thing in and out every day. And they were saying like the substrees change pretty much order by order. And they're like, well, hang on a minute, I wasn't picking this substitute five minutes ago. And so they were overriding the personalized recommendation and choosing the thing that they picked 10 minutes ago.
So that was obviously, we learned a lot from there. We learned a lot about how pickers use that. You know, we've got hundreds of thousands of pickers in the operation, so it's pretty enlightening to see that kind of insight. So that springs to mind.
That's really interesting. Was there any user research or anything done beforehand to say if we're going to change the way these guys do their job, what an impact is that going to have?
Well, we did, but as you know, with user research, I mean, I love user research, don't get me wrong. But if you ask someone a question in the wrong way, which you get the answer you maybe don't expect or maybe you expect it. So we were asking pickers and they were telling us how they should follow the process, not what they actually did.
That was again another enlightening piece of analysis from that particular test.
It's incredible what kind of impact you can have on such a vaccine. Vast front line and well, customers from.
Not this cheddar cheese. Pick that other cheddar cheese.
Yeah, and some pickers know the order because they pick the same thing for that same customer every week. So they'll know that that's for Mrs. Biggins down the road from their sister in law.
They know these customers which again, we don't tap into that enough. It's something we want to do more of using that pickup or knowledge base.
How could you tap into that? Because again, you tread this really interesting space of digital sort of experimentation and in person. So how do you get into their heads?
Well, again, the short answer is we don't right now. But the technology hasn't really let us because of the way that we work with Walmart. What we want to do and what we have done in the past is try to give the pickers some way of feeding back what they're doing. So we can take that verbatim and take that sort of very rich data and try and populate things like those trees.
The problem is we kind of say, oh, give us this feedback in one hand and then on the other hand we say you got to pick faster, you got to get through, you got to get through this order. So they don't always think they have the time to do it, even though we do ask them for it.
Yeah, it's always a conundrum. You want to get people to be efficient but also stop doing what they're doing and tell you things.
One of the things, sorry, just one of the things in the subs engine is and it seems like such a logical thing where dietary requirements are. I'm sure Bob is listening intently. But why would you pick meat product as a substitute for a vegetarian? Ordered a vegetarian pizza.
Why would you pick meat product, things like that?
In actual fact, because we don't actually. We can't personalize for a household. It's very hard to personalize for a household. You have to personalize.
You can really only personalize for a user. But our customers are households most of the time. Obviously we've got some single, single family and single people family and things like that. But that's really challenging to get that kind of implicit feedback to build those models.
What I'd love to do is get more explicit feedback, but that in itself drives a lot of challenge because as soon as you ask someone tell me your dietary needs, if they get a meat pizza for a vegetarian one, they're going to be up in arms.
So yeah, there's quite a lot of really good opportunity there.
I think it must be quite difficult in the grocery world too. I have a friend of mine who works in the food waste side of the food industry and she's working with companies that try and solve the problem of how do we prevent or reduce food waste. And actually it's really difficult for companies who are creating the food or the grocery stores and things like that to predict how the food is going to be consumed. Because you can have a pretty good idea of how much food you can produce, but you really don't know how much is to going to be consumed because people inherently don't know what they're going to consume day in, day out.
So that must be quite challenging when it comes to your subs engine and things like that. Have you picked up on any signals in that area?
Yeah, well, I'll give you a real world example of this. So we obviously work with our suppliers. So Coke as an example, came to us one day and said we've got loads of research about when customers buy a 36 pack of cans of Coke. We know they're going to run out in three weeks.
We know it. We've got loads of. We've got loads of data to back this up. So we want to retarget these customers that have bought their 36 pack of Coke in three weeks with a new 36 pack of Coke.
And what do you say to that?
In the U.S. yeah, maybe they're in the U.S. too.
So we tried it out and well, invariably didn't do anything.
But that's like. No, it's right. It has to be right. This is what the data's telling us.
And it wasn't right.
Well, not according to the test we ran. This was a long time ago. So maybe that's changed in Covid, but.
Maybe Coke Zero instead.
Coke Zero. There's too many flavors, isn't there?
So many. Okay, well thank you very much for all of that. That was very enlightening. Sort of a view into ASDA and also experimentation in the physical space.
I am going to. Well, this is the second time we've done this right. And really enjoying being up here having chats and I love finishing off with a rapid fire round.
Rapid Fire Questions
Now, Connor has not seen these questions, so the name of the game is.
I didn't see the last question.
That was an impromptu one, you know, interviewing.
So rapid fire round, you haven't seen the questions? First thing that comes to mind, very little explanation. Okay. You don't need to justify it.
Just say what you want to say. Okay. Favorite customer. Insight gleaned from an experiment.
Geez.
God, this is really bad.
Some random.
Yeah.
Why have I got so many favorites?
Okay, because you chose them.
Most useful experimentation tool or platform?
Well, I'm shamelessly going to say sciencebach.
All right, well done, ScienceBike. What is the most abundant type of bias you see in experimentation?
Well, it's when people morph their own homework, basically.
Okay.
Common CRO myth. Question mark.
It's not just. Well, sorry. It's all about conversion. Right.
Okay.
CRO job or skill or both?
CRO, I think. Sorry?
CRO, I think is.
I'm going to go off.
It'll be medium fund around.
CRO is skill. Experimentation is a mindset.
Oh, we've got some believers in the front here. Look at this.
Excellent answer. Okay. Centralized or decentralized?
Horizontal.
Feels a bit fancy. Okay. Stats important or not important? Important.
Most basic skill for a CRO professional.
You're not always right.
The term CRO. Hate it or love it?
I don't use it.
Okay, that answers that question.
All right, and our last question.
If you put up a Christmas tree, when should you put it up?
After my first daughter's birthday, which is 23rd of November. Although I can't this year because my house is a wreck.
Okay, so no sooner than the 24th of November.
Yeah, exactly.
All right, lovely. Can we give Connor a big round of applause?