Complex questions still need to be answered by people, not machines according to new research

Are online human-to-human questions and answers on websites like StackExchange being replaced by ChatGPT? According to researchers from Rotterdam School of Management, Erasmus University (RSM) the switch to using ChatGPT depends on the complexity of the question.

Dr Dominik Gutt and Dr Martin Quinn investigated how ChatGPT affected the number, complexity and novelty of questions asked using data from StackExchange, one of the largest Q&A platforms. The researchers classified users into three groups: casual, intensive, and top users, according to how often they asked questions on the platform in the past and investigated how the question asking behavior of these groups changed due to the release of ChatGPT.

The researchers found that questions from casual users have decreased the most, at 18%, but there’s less a drastic decrease in the number of questions from intensive and top users.

Interestingly, questions from casual users have become more complex, while the questions from intensive and top users have not changed at all in terms of their complexity and novelty.

“This suggests that causal users might just delegate easier questions to ChatGPT but ask more complex questions on the forum to be answered by humans. However, intensive and top users do not show this behavioural pattern. One reason could be that casual users are mainly looking for answers as such, while committed users value the community experience.” says Dr Dominik Gutt.

For those that rely on these Q&A platforms, this research offers an important insight: AI chatbots aren't necessarily killing these communities, but they are changing who participates and what gets discussed.

"But on the flip side, if the number of questions posted on public platforms decreases and the number on proprietary platforms like ChatGPT increases, then we lose publicly available knowledge," says Dr Martin Quinn.

This trend also helps GenAI developers who use data from Q&A platforms to train their models, such as ChatGPT. If the data they use for training improves in quality, then so do their models. These findings enhances our understanding of GenAI usage, it enhances the Q&A platform’s understanding of how it is affected by GenAI, and it may enhance the training data of GenAI models, which might increase GenAI quality and thereby benefit society. This helps us understand how AI tools might complement rather than completely replace human knowledge-sharing communities.

/ENDS

For more information, or to speak with the researchers, contact Kate Mowbray at BlueSky PR on Kate@bluesky-pr.com or call +44 710022871