Understanding the potential epistemic impact of sycophantic AI is an important challenge for cognitive scientists, drawing on questions about how people update their beliefs as well as questions about how to design AI systems. We have provided both theoretical and empirical results showing that AI systems providing information that is informed by the user’s hypotheses result in increased confidence in those hypotheses while not bringing the user any closer to the truth. Our results highlight a tension in the design of AI assistants. Current approaches train models to align with our values, but they also incentivize them to align with our views. The resulting behavior is an agreeable conversationalist. This becomes a problem when users rely on these algorithms to gather information about the world. The result is a feedback loop where users become increasingly confident in their misconceptions, insulated from the truth by the very tools they use to seek it.
Complete digital access to quality FT journalism with expert analysis from industry leaders. Pay a year upfront and save 20%.
，更多细节参见PDF资料
Go to worldnews
Турция сообщила о перехвате баллистического снаряда из Ирана14:52
'The AI Doc' trailer: Sam Altman, Dario Amodei, and more discuss the precarious future of AI