Recently, controversy and accusations have followed artificial intelligence companies flexing their models’ research capabilities. When OpenAI claimed an internal model had solved one of math’s seven million-dollar problems, a mathematician at New York University insinuated that the company had stolen his progress, which he had shared with one of their chatbots. Weeks later, the New York Times reported a similar story unfolding between Anthropic and a biologist at the University of Copenhagen regarding a group of enzymes.

Despite this, scientists from all disciplines are opting to use these companies’ large language models. Some use them as a soundboard or a tool for literature review, others for cleaning up nearly finished work. And mathematicians and theorists are tapping their new, powerful models to generate entire proofs for their research. But as researchers feed more of their data and ideas into these models, concerns are growing about how that content is getting used.

Both OpenAI and Anthropic stated unequivocally that they did not train their models on the mathematician’s and biologist’s uploaded data, respectively. But the companies—which have not consistently met academic norms—don’t have every scientist’s trust. Some suspect they may not even have the full picture themselves. Their agents could be accessing data in ways the companies don’t even realize. After all, it took three months for the public to learn that OpenAI’s models had breached containment and hacked into foreign government databases.

On supporting science journalism

If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

With these conspiracy theories swirling, some researchers are now seeing the models more as competitors than tools—and the use of them as a betrayal. “I’m not encouraging it, and I’m not doing it myself,” says Clément Canonne, a theoretical computer scientist at the University of Sydney. “I’m not using them.”

Ahead of taking their companies public, OpenAI and Anthropic are increasingly stepping into math and science to show how capable their products are. But AI has sometimes felt less like a guest over for dinner and more like an unwanted visitor breaking and entering. If scientists feel that these companies are rifling through their every interaction, they may start locking them out entirely.

The lack of transparency invites these sorts of fears. “Right now, these systems infer things about you without telling you that they’re inferring things about you,” says Jennifer King, a fellow at the Stanford University Institute for Human-Centered Artificial Intelligence, who researches data privacy and policy.

King says the only way for researchers to really ensure their safety when using large language models at the present moment is to look at their account settings carefully. According to her, “enterprise” accounts—which represent a special deal worked out between a client company and the AI’s owners—seem to be safe. “But if you are on a personal account, all bets are off,” she warns. She describes how users on personal accounts need to explicitly opt out in many cases to prevent their information being used from training.

Plus, even if a company isn’t intentionally using your data to train their models, one of their many agents could always go rogue, infiltrating company data and hoovering up individual users’ conversations with their chatbots. While there is no evidence this happened in the case of any of the AI companies’ scientific discoveries, rogue events over the summer have left some extra concerned about who they trust with their data.

The trouble is, it’s hard to get much evidence at all out of the tech giants. “The companies are not transparent about these practices, so it’s really hard to pin down,” King says. That means experts can’t speculate about any particular case. But in general, “the companies spy on you and use that information against you,” says Bruce Schneier, a security researcher and activist.

The new AI models are very capable, including for research, and many believe they will help researchers find new results faster. So scientists may feel they face a choice: either use the models and risk leaking a discovery or don’t and risk someone who does publishing it first.

All this has created an uneasy landscape in which many scientists are left feeling they have to give up their privacy and work to keep up. “There is definitely a pressure,” Canonne says. “If you’re not using them, are you going to be irrelevant?”

Subscribe to Support Independent Journalism

Great science journalism requires human expertise, time, effort and creativity. And it costs money. That’s why I and the journalists here at Scientific American hope you’ll join our community.

When you subscribe, you are supporting staff and freelance journalists who are passionate about telling science stories that are true, important and compelling. Our editors and reporters are often experts in their fields, which means they understand the nuances of big discoveries and can untangle the breakthroughs from the hype. With a subscription, you are also supporting rigorous fact-checking to ensure the words we publish are precise and accurate. And you’re supporting original illustrations, graphics and photos that bring you closer to an advanced laboratory, an ice sheet in Antarctica or a space mission in orbit. You’re helping us craft other types of high-quality journalism as well: Our newsletters are carefully written, edited and curated by staffers you have or will come to know and love. Our Science Quickly podcast is based on original reporting, collaboration with editors and scientists and exacting production.

Subscriptions keep this engine running so we can continue to deliver thoughtful, rigorous and independent science journalism to you. In an era of viral misinformation, this work is crucial. If you value what we do, I hope you’ll consider joining us as a subscriber