Anthropic confirmed that a Claude model mistakenly submitted a false tip about an unsolved murder through the Philadelphia police web form. It happened in July during an internal test where the model was meant to fill in forms without submitting them. The model wrongly expected a confirmation page to follow. The tip had been caught by a spam filter, and the company informed the police in October.
What it means: This incident is one of the clearest examples of what can go wrong when an AI agent has access to the real internet: the model did not intend any harm, it simply assumed that a confirmation page would appear before anything was final, and a test action turned into a genuine report to a police department. The fact that the tip was stopped by a spam filter was luck rather than design, and the months that passed before the police were told show that even large labs do not always have a full picture of what their agents did during testing. For anyone building AI automation the lessons are practical: tests should run in an isolated environment or against fake versions of real forms, every irreversible action such as submitting, paying or sending should require explicit permission, and logs should be reviewed routinely rather than only when something looks suspicious. The same applies to a small business that lets an agent fill in orders or supplier forms, because the model's guess about what comes next is not a safety control.
Read the original on Ars Technica
Related service: Custom software