Mellman: Different questions, different projections

Two different questions—both seemingly reasonable—yielding opposite forecasts. How is that possible?

Last week we warned about misleading crosstabs. This week’s warning is about different questions on the same topic producing different answers and divergent interpretations. 

Compared to forecasting elections you’d think it might be easy to determine whether people are going to spend more or less this holiday season.

It turns out to be rather difficult. So difficult that two different questions on the same topic yielded not just different answers, but contradictory interpretations. 

Gallup offered an upbeat assessment. Last week they unveiled poll results showing Americans not only predicted they would spend record high amounts this season, but also revised that estimate upward in November—only the fifth time in 18 years that’s happened.

Gallup based their conclusion on what appears to be a straightforward question: “Roughly how much money do you think you personally will spend on Christmas gifts this year?"

The average response? A record $975, $108 more than a year ago. The next highest total was $886 in 2021, while the low of $616 occurred during the 2008 financial crisis. 

Gallup supplied retailers with an optimistic scenario for shopping this holiday season.

Contrast that with last week’s Monmouth University’s survey headlined “Increasing number this year report holiday spending cutbacks.” 

Monmouth employs a different question in its analysis, “Have you cut back your shopping list this year because of high prices or are you purchasing the same amount of gifts as usual.”

Monmouth only has data from the last three years but the 55 percent who say they are cutting back is the highest in the brief series. The 37 percent who predict they’ll cut back “a lot” is nearly twice as high as it was in 2021.

While Gallup says spending is going up, Monmouth paints a grimmer retail picture.

Two different questions—both seemingly reasonable—yielding opposite forecasts. How is that possible?

First, the responses may be less different than they appear. 

Gallup is averaging together spending by individuals. Wealthier people, a minority, spend more—an assumption Gallup’s data confirms—and the wealthy plan to increase their spending most. 

That could increase average spending even while the larger number of lower income people reduce their spending—which Gallup’s data also confirms. 

The two pollsters don’t release the data required to do the arithmetic precisely, but it’s entirely possible both are correct. Average spending is going up (as per Gallup) because the rich are spending a lot more, even while a majority is spending less (as per Monmouth). 

Careful reading reveals other differences between the questions.

Gallup asks specifically about spending “on Christmas gifts.” Monmouth asks about your “shopping list.” 

If your shopping list last year included a new winter coat you bought yourself, you could decide not to count it among your Christmas gifts. Spending on your “shopping list” could decrease while your outlay for “Christmas” gifts might increase. 

That could account for the disparity between the responses.

In addition, Gallup asks about dollars spent, Monmouth implies its question is about the number of gifts purchased.

Another distinction is Monmouth’s direct invocation of “high prices” as a reason for cutting back spending. That reference alone could make people more negative in projecting their holiday spending.

Alternatively, perhaps people are just not very good at forecasting their holiday spending. In theory, multiplying Gallup’s average forecast by the adult population should come close to actual holiday spending. Based on data from the National Retail Federation, respondents’ prediction was off by a massive $700 billion in 2022—a 300 percent underestimate. 

It’s not intuitively obvious which question is the superior measure.  

The first step in making that assessment would be understanding the relationship between each of the questions and real-world outcomes, difficult here due to the paucity of data.  

Moreover, it may well depend on what kind of market a particular retailer serves—those who buy gifts at Hermes may spend differently than Walmart shoppers.  

Note also that getting a full picture may require multiple questions instead of simply relying on one. 

The key lesson though—poll questions don’t interpret themselves. They require careful reading, deep analysis and, to employ a word given a bad rap by college presidents, historical context. 

Mellman is president of The Mellman Group and has helped elect 30 U.S. senators, 12 governors and dozens of House members. Mellman served as pollster to Senate Democratic leaders for over 20 years, as president of the American Association of Political Consultants, a member of the Association’s Hall of Fame, and is president of Democratic Majority for Israel.