One of the greatest MMM experts in the U.S.

Maziar Nodehi at interviewing Anurag Joshi from Facebook, one of the greatest MMM experts in the U.S.

A Decade Dedicated to Data

There are different ways to describe a community. The most common description focuses on certain commonalities, such as norms, religion, values, customs, or identity.

But whilst this is certainly true in many circumstances, I feel this description is missing something fundamental. And that is shared experiences. One such experience is having seen the inception and birth of modern marketing analytics. And in particular modern Marketing Mix Modelling (MMM).

There is a global network of highly-skilled, dedicated MMM professionals who have mastered their trade through a unique combination of undying curiosity, relentless problem-solving skills, and an analytical mindset. These people are far, and few between, and highly sought after in the industry.

Meet Anurag Joshi, a resident of San Francisco, California. Having dedicated nearly eleven years to helping organizations use numbers to better themselves, he recently landed the ultimate job at Facebook, helping customers make the most of the tech giant’s data platform. And yes, he is guiding clients who use MMM as the methodology. 

The long road to 1 Hacker Way

According to an old Chinese proverb, every journey starts with a simple step. Anurag’s journey began with a few simple steps, graduating from College of Engineering Pune in 2008.

From there, a move to the University of Buffalo to take on his first real, hands-on challenge developing optimization algorithms and researching data mining and graph matching. 

Little did he know that his decision to leave Pune behind would lead him to a key role at Facebook, shaping how businesses do modern marketing.

I met Anurag over a Google Meet call in mid-December. At the time, he was working from home because Facebook’s offices were still closed due to Covid-19. But this didn’t seem to phase Anurag. He seems content and focused on his work, and he greets me with a warm smile.

Given his decade in the industry, I’m inquisitive about hearing what key insights he has for anyone just getting started with MMM.“Well,” Anurag begins, observing me make sure I’m paying attention.

At the heart of everything is data. And depending on the industry you are in, the process of gathering marketing data is different. As an example, the way you collect data in the retail space is different from how healthcare/pharmaceutical companies gather data.

“Industries have different external factors affecting their sales performance. For consumer goods, you have many variables, from seasonality, gas prices, and unemployment rates, to things like brand perception and brand equity. External variables include things like new FDA approvals, publication, or an upcoming clinical enable additional line extension in the pharma world. If a drug were to be approved for a new age group, for example, that event would trigger higher sales. ”

“Secondly, the way the models are built can be very different. There are two primary techniques: some use a Frequentist approach, while others use a Bayesian approach. Similarly, you can have models that are additive or multiplicative in nature. In the end, we need to remember that MMM is a regression model. ”

One thing to remember is that with regression you are still measuring correlation; not causation. So with MMM, and all the inputs that it gives you, you’re still always measuring some kind of correlation, and that can be a challenge.

“The true impact of your advertising can only be measured using randomized control trials or RCTs. The concept is similar to what is used in medicine. You have a test group (group exposed to advertising) and a control group. Using RCT, you can assess the incremental impact of your advertising. 

The surrounding world, however, doesn’t always subscribe to these attributes. Sometimes the data you manage to get your hands on is incomplete, and sometimes there is a real risk that the data will lead to misguided decisions. ”

“Every project related to MMM has been a great learning experience, with many success stories. Sometimes you get precious insights that go against preconceived notions.

I’ve been in situations where the hypothesis is that the advertising won’t lead to revenue, but then we see that advertising has a solid ROI. I’ve seen things go the opposite direction as well when the prevailing belief was that advertising was driving a lot of the revenue, but the MMM study shows that’s definitely not the case.”

Anurag’s advice for anyone running regression models is to run a randomized control trial simultaneously.

“Let’s say you are the Chief Marketing Officer and you decide to launch a new product. To support this launch, you decided to run a television advertising campaign. The product sells like hotcakes because it’s an excellent product and people love it. But as the CMO, you might think:

‘You know what, my sales are going up because I launched a TV campaign. I’m going to increase my budget for television advertising.’

As the CMO in this situation, you would be forgiven for putting more money into television because your sales keep on going up. It’s only natural for a CMO to assume causation in a case like this. It’s also highly likely that the running model will show a correlation of sales revenue with television advertising. And based on that model, the traditional recommendation would be to increase the investment in television further.”

“This is why I recommend running a randomized control trial to scrutinize the validity of your model,” Anurag continues.

“You need to launch the television campaign, but instead of running the television campaign in all designated markets, only launch the campaign in half ( of them or at least some markets). Keep the other designated markets in the dark. Always look at both the results from the regression(MMM) model and the randomized control trial. This is the gold standard for separating causation from correlation.”

As I started to thank Anurag for this time, he couldn’t resist adding one last detail.

“This might be going a bit deep, but I also want to point out the biggest challenge in just doing a randomized control trial. That is the fact that you’re measuring the last dollar incrementality. Let me explain. Say you are running four channels in a given country, and let’s say you introduce an additional channel – channel Y. Then you run a randomized control trial. And you see that wherever you are running all five channels, you are bringing in $100 in revenue. However, in the areas where you are running only four channels, you’re only getting $90. Which means that adding channel Y produces $10 of value, right?

Wrong. You can’t just stop all channels except Y and run it expecting to yield $10 of revenue.

What the last dollar incrementality means is that the addition of a fifth channel versus running only four channels yields an incremental benefit of $10.”

This is the sort of impassioned insight I hear a lot from MMM masters. They have seen how the analytics can reveal truths the way daylight, in its path across a room, sheds light on corners you didn’t realize were getting dusty. Once the eye has seen the flaws, you cannot unsee them. You have to grab the vacuum and get to work. 

Written by: Maziar Nodehi, Proof Analytics