斯蒂芬·韦德：彩票市场营销效果评估的思路

 斯蒂芬•韦德 Stephen Wade

 

Common-Sense Construction

 

Stephen Wade, Research and Development Manager, Washington’s Lottery and Principal, Lottery Management Consulting, LLC

 

I am recounting the story of a consulting engagement where my role was to help measure the effect of advertising, on sales of the big bloc games. We were meeting in the stylish offices of the advertising contractor. I had been introduced to some of the principals on the account, at least two of whom happened to have been professional models earlier in their careers. They were flattered to think that I knew this because I had researched their backgrounds before coming to meet them. They had understood me to claim that I had been a model, too.

 

Of course, what I had meant to communicate was that I was an old hand at modeling of the mathematical sort. I was amazed by the misunderstanding. I did not recall anyone ever suggesting that I am photogenic. However, it was working well for me. I was doing my best to keep my unearned positive halo, while honestly answering their questions.

 

My method was to avoid any further reference to modeling, mathematical or otherwise.

 

In this style, we had already dealt not only with the fundamental dynamic of the work - I predict how ‘no-change’ will look, we test my predictions against reality- but with how that dynamic made us feel. It made us feel like we were on opposing teams, in a friendly game.

 

Veronique, whose beauty was said to be documented in a lingerie catalog from 1985, moved us back to practicality with the question,

 

“And how do you go about making these precise predictions?”

 

I had made big claims for the value of defining predictions tightly, and now I felt like I needed to show how my methods made sense.

 

“First of all,” I said, “I try to build a math framework that corresponds to a common-sense narrative. That way, when we’re done, we can talk about it and make sense to one another. We would not have to do that – I could just be satisfied when the numbers come out close to reality – but I believe starting with what we say about the games helps us in the long run.”

 

Jade, who had worked only a couple of years ago for something called “Title Nine,” was skeptical: “So at some point we’re going to be talking common sense, and you’re going to be talking math, and we’re all going to be better off?”

 

“Let’s try it and see. What would you say about how sales in Mega Millions go?”

 

“The jackpot builds from one week to the next, and sales go up with the jackpot. Sometimes the jackpot gets to a certain size and then everyone goes nuts, and sales go through the roof, and then it gets won and we’re back where we started.”

 

Looking to the lottery people I asked, “Anything else?”

 

Andrew, my main contact at the lottery, spoke up: “We know that when everyone starts going nuts, we get people buying tickets who normally don’t. That is, they may have played before, but only the last time the jackpot was big enough to cause a buzz.”

 

“And how about the people who do normally play – do they spend that same regardless of the jackpot?”

 

“Well, no. They’ll spend more too, as the jackpot gets bigger.”

 

“And do sales always go up as the jackpot goes up?”

 

I heard a mixed chorus of “Yes” and “No,” and looked again to the lottery people.

 

“When the jackpot grows from a Friday to a Tuesday draw,” Andrew said,” it’s actually pretty common to see sales lower on the Tuesday.”

 

“OK now,” I said, “What you have just told me is enough to let me frame up a good math description of what goes on. There are people who play all the time, and increase their spending as the jackpot grows, and there are people who come in only when the jackpot has grown so big, and play like they’re excited. And it’s different on Tuesday than it is on Friday. My job is to put that into math expressions, and then go to the lottery for all the draw-to-draw sales data for the last year or so. I can use that data to make this description precise. By that, I mean quantitative. For instance, I will come back to this conversation with an estimate of just how big the jackpot is where people get excited.”

 

“But we already know where that is,” said Brad from the lottery. “It’s about $200 million.”

 

“If not $300,” countered Andrew. “And is this threshold the same for Tuesday and for Friday play?” I asked. No one seemed to feel strongly about this.

 

 “So, you talk to people and they vote on where they think the threshold is?” asked Jade.

 

“No, I talk to people and verify that most think there is a threshold. Then I make sure that idea of a threshold is part of the mathematical description I eventually write. Also, people agree that sales grow as the jackpot grows; I make sure that gets captured in the math. Also, we see that even though the jackpot may grow from the weekend to the middle of the week, sales are sometimes lower in the middle of the week, even for the bigger jackpot. We make sure that can be part of the story, just by providing some math expressions. We set up the math to say: sales could depend on the jackpot, and which draw day it is, and how big the jackpot is compared to the threshold, and by the way some people may play only above the threshold. That’s about it. “

 

“That sounds pretty general.”

 

“I hope that it is. I try to frame up the mathematical description so it could fit lots of different player behaviors. Flexibility is key. There are seven points of adjustment that we work on, to try to get the story coming out of the math to line up with what actually happened. Once we get the math to recap the actual history, we say we are ready to predict the future.”

 

I looked around the room. Everyone still seemed to be present. However, I was pretty sure I was talking too much.

 

“I can tell you more about how we decide we have the best possible fit, if you like. I can tell you about how we do the adjusting. Or I can tell you about how we run the prediction forward, and test whether it still works.”

 

Veronique looked around the room. “It’s true that we have a lot to talk about today,” she said. “Maybe we can move ahead to the part where we test whether your model still works, and hopefully show that it doesn’t. As for the rest of it, I assume you’re using a nonlinear solver, and minimizing squared deviations?”

 

This question from Veronique gave me surprise, then delight, then embarrassment, all in the space of less than a second. Surprise, because the question implied a deeper knowledge of this area than I had assumed of her. Delight, because I knew she would understand and appreciate what we were doing. And then embarrassment, because I knew my surprise might have been less, had Veronique been plainer-looking.

 

“That’s exactly right,” I said, doing my best to avoid expressing too many of these various emotions. “And I fit both a coefficient and an exponent for a jackpot term, for each player population.”

 

Veronique smiled. She had been playing me! If anyone in the room was both a model and a modeler, it was Veronique. But she also had been helping me. She evidently saw the value in keeping the discussion jargon- free. Understanding that I could count on her, I went on.

 

“So, going forward, we will have our story and we will make our predictions of what sales will be, one draw at a time, for several weeks. Meanwhile, you are doing what marketers do. Players are doing what they do. After some time, we ask: have the players been doing anything different lately? That is, are actual sales different from what we predicted?”

 

“Now, it could be that they are doing something dramatically different. Maybe they are buying 30% more wagers than they would have done under the status quo. In that case, we could look at just about any individual drawing and probably see a difference that would pass a test for reliability. That does not happen very often.”

 

“More likely, the players are doing something a little different. On some drawings, the actual sales are a little higher than predicted, on others maybe not. Is this a real change? You could not prove it on any individual drawing. Instead, we look for consistency of direction. If we look back at the record that we “trained” our predictor on, and compare the real sales history to the story produced by the math, we usually see that the math missed in one direction about as often as in the other. That’s to say, we hardly ever got it exactly right, but we over-estimated about as often as we under-estimated. That’s a good thing, and we expect that to continue, so long as our prediction continues to work. The first sign that we have left the status quo behind is usually that the errors become very consistent in direction. Once we are satisfied that the consistency is not merely due to chance, we can estimate the average size of the change over many drawings.”

 

“So how big does the change have to be, to be recognized this way?” asked Jade.

 

“If the change is very consistent and lasts for a long time, it does not have to be big. We have documented shifts of about 5 percent this way.”

 

People seemed to be relieved by this number. I realized that their feeling was sort of like the one I get on those rare occasions where some necessary thing turns out to cost less than I had expected. They seemed pleased with me. My unearned positive halo was intact, and I reflected that everyone was probably ahead when I explained my work without using the expression, “mathematical model.”

 

In the next article, I will relate how Veronique and Jade surprised me when I sat down with this group again, a few weeks later.