The Business Research Scam No One Talks About: What should you believe?

Episode #357 The Business Research Scam No One Talks About: What should you believe?

CAN WE BELIEVE BUSINESS RESEARCH?

Last week, I explored the balance between aspirational ethics and some of the more challenging things you may have to do as a leader.

I picked on a piece of research from Harvard Business School, which concluded that investors don’t support certain management decisions… like laying off staff to improve company performance.

Apparently, the study showed that investors are more high-minded in their ethics than we give them credit for. People before profits, right!?

Many years ago, I would’ve taken this on face value and said to myself, “Wow, that is so interesting! I’d better factor that into any future structuring decisions I make for my team.”

These days, though, I question everything… and with good reason – in the last several years, there have been many high-profile cases of alleged academic fraud: fabricating data, stacking sample groups, and gaming results.

In this newsletter, I take a deeper dive into the gilded world of business research, and ask the question, “Is our faith in business research sometimes misplaced?”

I start by exploring how business research has evolved over the last 100 years or so; I bring you up to speed on the latest efforts to expose fraud in business research; and I give you three ways to use business research to best effect, without blindly accepting the findings.

THE ORIGINS OF ACADEMIC RESEARCH IN BUSINESS

I’m not sure if I should read anything into this, but when I was doing the background work for this newsletter, I reached out to two colleagues of mine, both of whom are highly respected senior academics in top global business schools…

Tumbleweeds!

… but I’m going to push ahead anyway.

For over a century, we’ve held a deep fascination with research from the world’s top business schools. At the beginning of the 20th century, Frederick Taylor’s theory of scientific management started a boom in business research.

Taylor proposed that any work that was done in a company should be scientifically structured, measured, and reviewed. He was convinced of the principle that management’s job was to put processes in place that could be readily followed; workers were hired simply to execute those plans efficiently.

In the hundred-odd years since Taylor’s ideas took root, countless management theories have been proposed, studied, and documented, in areas as diverse as strategy and organizational behavior; operating performance; negotiation; and – dare I say it – leadership.

Over time, the research shifted from quantitative business disciplines, like economics, towards the behavioral sciences.

Findings from academic research make their way into the mainstream through business books and articles, which are published in respected periodicals like the Harvard Business Review. Many leaders in the corporate world treat these studies with the same reverence as the tablets that were brought down from Mount Sinai.

Over time, a thriving industry sprang up. Business schools richly rewarded the academics who asked the most insightful questions, and then tested their hypotheses by undertaking rigorous studies, which – somewhat predictably – prove their theories to be right.

Business school academics saw the commercial potential of their research. They would be handsomely rewarded if they could confirm their groundbreaking theories: lucrative speaking engagements would follow; as would the book publishing deals; offers of seats on Fortune 500 boards; and highly paid tenures at the top business schools.

Rising to the top guaranteed fame, fortune, and widespread adulation.

But let’s think about this from our perspective as the consumers of the latest business and leadership wisdom.

For me, especially when I was younger, I would take any academic research on face value. I would automatically assume that all research was thorough, free from bias, statistically valid, and founded in a solid research methodology.

Any framework I came across in one of my MBA textbooks… any article that was published in a reputable journal… any principle that was revealed in a highly acclaimed business book… any new finding heralded by a rockstar academic….

I didn’t question it! I would try to find a way to integrate the findings into my own leadership approach.

But, as time went on, I became more and more skeptical. The field of academic research is highly competitive, and with the incentives on offer, very prone to gaming… but more on this shortly!

HOW GOOD IS “GOOD TO GREAT”?

Criticism of academic research methods isn’t new. One of the seminal business publications of the 21st century is Jim Collins’ book, Good to Great. This book has no doubt been used by millions of leaders in the quest to find the secret source for creating elite business performance.

But it’s also been criticised, quite rightly, for some obvious shortcomings in the methodology that Collins and his research team used.

The first is selection bias. The companies that Collins studied weren’t chosen randomly. They were selected on the basis of past performance. Of course, this is entirely reasonable given the nature of the study. It just meant that there was no control group to compare the true effect of the identified success factors.

The second criticism is attribution bias. Looking backwards to attribute success to certain factors is prone to rationalisation and nostalgia. Why were we successful? Well, obviously because of our unique talent, and a culture of relentless performance. It had nothing to do with privileged assets and market fundamentals, right!?

The third criticism is narrative fallacy. We naturally connect the dots to explain historical events in story form, filling in any gaps to make sure it’s consistent. The end result is that the narrative is made to be more compelling – it sounds like it should be true.

Despite all of this, Good to Great remains one of the classics. It resonates deeply because it reveals some powerful principles. These principles make sense and what’s more, many of them actually work when you apply them in the real world.

In the same vein as Good to Great, Malcolm Gladwell is roundly criticised for his lack of academic rigour. But Gladwell is a compelling storyteller. He’s a really creative thinker, and a brilliant student of the human condition. Outliers is still one of my favourite books.

Is there value in reading Gladwell’s work, and adapting certain principles for your own leadership approach? Of course!

There’s no doubt that some works are more storytelling than science, and that’s absolutely fine. Since the dawn of time, we’ve learnt through storytelling. We naturally integrate the lessons into our personal experience – this is how we discover meaning and context.

As long as we’re mindful of how much validity we can ascribe to anything we read, see, or hear, then we can apply the lessons judiciously.

The real problem arises when business research is positioned as being empirically sound, but the results have been fabricated. We place more faith in academic research than information we get from other sources. So, you might be as shocked as I was when I learned how much of this research is subject to manipulation and fraud.

AN EXPLOSION IN ACADEMIC FRAUD

Well-meaning errors and innocent oversights in academic research are very different from deliberate acts of manipulation and fraud. In an industry where success is determined by the volume and uniqueness of research, the incentives to bend the rules are high.

Publication is the currency of advancement.

To piece together the state of academic fraud in business research, I relied on two main information sources. The first is an article from The Atlantic titled, The Business School Scandal That Just Keeps Getting Bigger. And the second is an episode of the Freakonomics podcast, Why is there so much fraud in academia?

Neither of these involved formal statistical analysis, but they have some super interesting findings. Fraud in academic research isn’t exactly a new phenomenon, but now a lot more of it is being identified and exposed:

• In 2002, only 119 academic papers were retracted due to faulty or manipulated data and methodology.
• In 2022, over 5,500 papers were retracted.
• Last year, over 10,000 papers were retracted.

How long do you reckon this has been going on for? Was it always this way and now people are just starting to dig to find the anomalies? Or is the number of fraudulent research papers increasing?

When I look at this growth in retractions, I wonder if it’s the result of rationalisation by the academics who produce the work. Just like performance enhancing drugs in sport, perhaps they think, “Everyone else is doing it, so if I don’t cheat, I’m actually disadvantaging myself.”

I’m just speculating here…

My fascination started with an article I read a few years ago about a group of researchers who go by the catchy name of Data Colada. They started to investigate potential fraud in research papers and found a staggering number of studies had been gamed, manipulated, or modified in order to prove their hypotheses.

To show how easy it is to fabricate research findings, the Data Colada team conducted a study where they intentionally manipulated the data. They released a paper titled, False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.

They managed to cherry-pick the data to conclusively prove the hypothesis that if you listen to The Beatles song, When I’m Sixty-Four, your age will be lowered by one and a half years. Clearly ludicrous!

Some of Data Colada’s biggest takedowns have been highly acclaimed academics, Dan Ariely and Francesca Gino.

Both Ariely and Gino have vigorously denied the allegations.

But the evidence, as presented by Data Colada seems fairly compelling, and the allegations leveled at Gino’s work in particular are rich with irony.

She forged her reputation as an expert in – wait for it – morality! Gino specialises in why people break rules and commit fraud. In fact, she produced a research paper on the human tendency to misreport facts and figures for personal gain.

She’s accused of faking data for a paper about why people fake data. I sh!t you not!

One academic, Juliana Schroeder, decided to investigate the allegations herself, after some of the papers she had co-authored with Gino were brought into question. Schroeder found that a 2016 paper that she co-authored with Gino and five others had serious irregularities in the data.

It looked like the data had been “made up”.

As one of the few insiders to take this problem head on, Schroeder obviously understands that the best disinfectant is sunlight.

A BIGGER PROBLEM?

What motivates someone to commit academic fraud?

Is it the blind ambition to succeed that entices rule-breaking?

Is it the belief that they won’t get caught or that the sanctions if they do get caught are worth the risk?

Is it just the ratcheting-up of the aha! factor, where the pressure to produce ever-more impressive findings pushes academics to resort to manipulation?

Dennis Tourish explored this in his 2019 book, Management Studies in Crisis: Fraud, Deception and Meaningless Research.

Tourish observed that the explosion of management research coincided with a simultaneous erosion of trust: replication failures, fraud, retractions and the growing gap between theory and real-world practice started to chip away at the previously unquestioned efficacy of the research.

Tourish argued that rising retraction rates were a reflection of systemic flaws in the peer review and editorial processes.

Business schools, journal editors, and peer reviewers are interested in protecting their own reputations, which is one of the main explanations for the fact that so few people who have serious irregularities in their research are brought to account.

Tourish says that research has evolved to a point where dubious statistics are used to prop up weak theories, especially in leadership.

And one of the blindingly obvious factors that isn’t mentioned much? In research teams, there doesn’t really seem to be any single-point accountability.

In co-authored papers, the most acclaimed academic’s name normally appears first in the list of contributors… but who was accountable for ensuring that the data is accurate?

In many of the cases where data manipulation had clearly taken place, the original data collection results were no longer available… what’s more, no one could seem to remember who had collated it, who had analysed it, and whether or not the principal authors had reviewed it.

It could well be the case that Ariely and Gino were absolutely on the level, but didn’t exercise sufficient control over the research process to prevent fabricated data from making its way into the studies. I guess we’ll never know.

If this was something as simple as a drunk driving test, it’d be thrown out of court due to basic flaws in the chain of custody of the collected sample.

And here we are, placing our stock in academic research as the ultimate source of truth, using it to guide our perspectives, and to influence the leadership decisions that we make.

THREE WAYS TO GET THE MOST FROM BUSINESS RESEARCH

I think we can all agree that we shouldn’t believe everything we read, hear, or see when it comes to business and leadership research… but we don’t want to throw the baby out with the bathwater.

Surely, there has to be some value there, somewhere?!

I must say, I’ve gained enormous value from exploring the latest research widely, whether it’s formal academic studies or informal surveys taken by reputable brands like McKinsey, cultivated from their direct access to some of the world’s top executives.

How do you work out what’s true and what’s not? Here are three things that you can do:

1. Check whether the principles coincide with your experience.

The very first thing is to carry out your own reasonableness test. For example, the “happy workers are productive workers” theory never quite sat right with me. I’d seen too many people who happily went about their work, yet they were obviously unproductive or otherwise ineffective.

As I explored in last week’s episode, the finding that “investors oppose layoffs, despite the potential detriment to profits” didn’t coincide with my broad experience dealing with investors in a wide variety of commercial situations.

I simply didn’t believe it… so I took a deeper dive into the research methodology to see if it was likely to be true (or not).

The danger with taking this approach, of course, is confirmation bias. Are you just looking for theories that prove something that you already believe? It’s really just about using your professional judgment and experience. When you read something, ask yourself the question, “Does that feel right?”

2. For anything that does feel right, does it translate into your context?

If any finding seems like it could be useful, work out if it’s true in your specific circumstances. Every situation is different, and many things that are generally true might not be true under certain conditions.

For example, the Hawthorne Effect was a phenomenon observed almost a century ago. People perform better when someone is observing their work. This just seemed to ring true with me.

But I didn’t start to blindly apply it. I ran my own mini-experiments to determine the extent to which it might be true, and under what circumstances.

This enabled me to work out how to optimise the principle based on my own style, my team’s capabilities, the culture of the organisation, and the type of work being performed.

3. Learn to use AI.

Interestingly, I use ChatGPT more and more to help me investigate things. I especially like using it to ‘research the research’.

For example, let’s say I came across an article in a magazine that said, “Taking a vitamin C supplement is good for you.”

Would I start taking Vitamin C?

Well, not until I’d done a little bit of background work… so, I’d start a ChatGPT to see if there’s any evidence that this might be true. I’d drill down and ask a bunch of questions, like:

• Have any double-blind clinical trials been conducted?
• Is there any other research that produced contradictory or inconclusive findings?
• Has the research methodology been brought into question?
• Has the author been implicated in any research fraud, or previously had to retract any papers?
• Does dosage make a difference to the observed outcomes?
• What are the documented side effects?
• What’s the incidence rate of any side effects for my age and sex?

After all of that, then maybe – just maybe – I might decide that it would be helpful to start taking a vitamin C supplement.

Now, just imagine if you applied that type of inquiry to every piece of conventional wisdom that came into your field of vision… it’s very enlightening!

RESEARCH IS ONLY USEFUL IF IT’S APPLIED

There’s a huge amount of potential value in academic research.

At best, it produces insights that contribute directly to your ability to achieve better outcomes.

But when contrived, manipulated or gamed, it can give you a full sense of security about things that may or may not be true.

So, use your judgment, and use your experience. Don’t blindly accept or apply the conclusions. Instead, use it as a starting point for your own observations.

Even the most solid research can collapse under the weight of practical application in the real world. And, let’s face it, isn’t that the whole point?!