The Best Way to Work with AI? A Study May Reveal the Answers

François Candelon, managing director and senior partner at consulting giant Boston Consulting Group, was curious to see just how useful generative AI, the technology underlying ChatGPT, really is in a professional setting. Candelon worked with some of the world’s leading universities to design an academically rigorous experiment putting generative AI to the test. And he used BCG’s own consultants as subjects in this exercise in social science.

Candelon’s findings may shed some light on how all sorts of companies and professionals can get the most value out of generative AI.

Here are highlights of the conversation that aired on WSJ Pro’s Executive Insights, a podcast that goes inside the C-suite with essential context about industry news and trends.

WSJ: What inspired this study that you undertook?

Candelon: We thought that a lot was said on generative AI as a technology, but just a little was said about the question of the adoption. When should you use it or not? What is the impact on your people’s strategy? How will it redefine your workflows and the human and AI collaboration? We wanted to focus on the human part of it, on the business part of it.

WSJ: Generative AI in the past year has been studied from so many different angles. We’ve seen countless studies. What made this particular approach unique?

Candelon: So most of all, that it was really scientific. And this is why I wanted to work with some of the best scholars—from Harvard Business School, from MIT, from Wharton—to make sure that the way we were looking at facts was indisputable.

The second thing is the fact that it was quite large scale, more than 750 BCGers—large scale with real workers, real white-collar workers. And the last thing that made it unique, I would say, is that it was focusing on real tasks.

WSJ: Let’s rewind. Talk a bit about what it means to conduct a scientific survey as opposed to any other kind of study or survey?

Candelon: First of all, it’s an experiment. It’s not a survey. Second is the fact that we were extremely clear on the fact that we needed to have control groups…We used very advanced techniques as well, for instance on semantics, to be able to see the similarity of ideas, the difference of them, and so on. We worked extensively and intensively with scholars to make sure that we were doing it the right way.

WSJ: The subjects were given tasks?

Candelon: It was an actual problem for an actual client. Of course, we sanitized the data, but they were very realistic tasks.

So the tasks were the following. On one, we had something that we called creative product innovation. For this, you are to find at least 10 ideas for a new concept of shoes for underserved segments. Then, once you had that, you had to pre-introduce them to your boss. How would you pitch them to your boss? Then, you would have to create what are the key questions for a given focus group. And what…given these segments, what would the answers be? Last but not least, it was really working on the social media campaign.

How can you do this social media campaign? So it was really something that was on that front, that was on creative product innovation. We believed up front, and it was confirmed later, that this is something where gen AI would really support humans.

WSJ: What did you learn from this process?

Candelon: On the creative product innovation task, humans supported by AI were much better than humans. But for the problem solving task, [the combination of] human and AI was not even equal to humans…meaning that AI was able to persuade humans, who are well known for their critical thinking…of something that was wrong. It’s difficult for people to understand [whether AI] is providing value or not. For companies, it is important to understand where you can use it, where you cannot use it, by experimenting.

The ones [consultants who participated in the experiment] who are below average [in performance] were actually benefiting much more from AI, which means that for a company, it has plenty of implications for their people strategy.

WSJ: Is this counterintuitive in some way? I think that we’ve spent years listening to people tell us that AI will help people with repetitive tasks, with routine tasks, and that this will free humans up for higher order creative thinking and problem solving and innovation.

Candelon: What was said was not wrong, but we need to realize that we have, actually, with generative AI, a new right brain. It is something which is based on innovation, on ideas and so on. Analytical AI is a fantastic left brain, extremely logical… So we need to be careful not to say that AI equals generative AI.

WSJ: What is left for humans?

Candelon: There is this famous quote saying that humans won’t get replaced by AI. They will get replaced by humans using AI.

WSJ: Stepping back, what are the implications of all this research for companies and for businesses and for people in general?

Candelon: Data was already very important, but with gen AI it will become even more important, as there is a leveling factor from gen AI.

The second thing is that you will need to revisit your workflows end to end, and see how humans and AI should collaborate…I give you one example…Instead of maybe starting with AI and trying to improve those ideas with humans, maybe you should do it the other way around. Start with humans, with the diversity [of ideas], and then try to see how to improve these ideas with AI.

Very often people in companies are saying, ‘OK, how can I do this use-case? What can I do with gen AI?’ And I think this is a wrong question. The question they should ask themselves is, how can gen AI support me in solving a business question?

And if this business question is based on efficiency, [ask] how can I maybe, lower my cost base in this function?

So how can I re-engineer this function—marketing, customer service, legal, whatever—with the support of gen AI tools? Or it could be about effectiveness. How can I improve and get new ideas for my products? By solving these business questions…you will have a significant impact on your P& L.

Write to Steven Rosenbush at steven.rosenbush@wsj.com