Spooked by Mythos, Trump suddenly realized AI safety testing might be good

Without defining standards, “the process can be politicized,” Kreps said. That risks creating a system where “whoever holds power gets to shape how the vetting works.”

So far, neither the Biden nor the Trump administrations has figured out how to avoid that, Kreps said.

Fears of government controlling AI outputs

Microsoft’s blog said that “CAISI, Microsoft and NIST will collaborate on improving methodologies for adversarial assessments,” which suggests that the plan is to develop these standards on the fly. According to Microsoft, “testing AI systems in ways that probe unexpected behaviors, misuse pathways, and failure modes” is “much like stress-testing whether airbags, seatbelts, and braking systems work effectively and reliably in safety-critical driving scenarios.”

But Gregory Falco, a Cornell University assistant professor of mechanical and aerospace engineering and expert in tracking governance of AI, insists that there’s a better way.

“Government oversight of AI cannot simply mean political review of model outputs, nor should it become a mechanism for deciding whether a model says favorable or unfavorable things about a president or administration,” Falco said.

Rather than relying on a politicized government leveraging evaluations to control the AI systems that the public uses, the US could build “some form of independent audit,” Falco said.

Imagine, Falco suggests, if AI firms understood that their models could be audited at any point, how much more accountability and discipline might such a system create? Operating similarly to the Internal Revenue Service (IRS), a rigorous AI audit system could create “real consequences for reckless deployments,” Falco said. For AI firms facing such consequences, the pressure would be on to ramp up internal AI safety testing, Falco suggested.

That seems like the “only viable path,” Falco said, since “the federal government does not currently have the in-house technical expertise, infrastructure, or day-to-day insight needed to directly evaluate these systems on its own.”