A Judicial Wake-Up Call on Government by AI

Perspective

A Judicial Wake-Up Call on Government by AI

Jordan Ascher / Jul 2, 2026

Protesters against the Trump administration gather in front of the US Capitol in Washington, D.C., on April 5, 2025. (The Yomiuri Shimbun via AP Images )

Republish

The Trump administration was all-in on using AI to slash federal regulations before the president’s second term had even begun. In November 2024, Elon Musk and Vivek Ramaswamy laid out a vision for AI-powered deregulation, and since January 2025, DOGE (the Department of Government Efficiency) and other federal agencies have sought to use large language models to undo swaths of regulations with barely any human oversight. This maximalist strategy would sacrifice the careful work of federal regulators for the sake of speed.

In May, a federal court dispensed the administration a dose of reality. The case, American Council of Learned Societies v. National Endowment for the Humanities, concerned an initiative closely related to DOGE’s deregulatory push: its mass cancellation of NEH grants in 2025.

In litigation, an unsettling story emerged. Shortly after President Donald Trump’s inauguration and his mandates to terminate funding for “diversity, equity, inclusion, and accessibility” and “gender ideology,” two young DOGE staffers arrived at NEH. They fed descriptions of hundreds of NEH grants into ChatGPT with the prompt, “Does the following relate at all to DEI?”

The chatbot produced bizarre outputs. For example, it flagged a grant to a whaling museum in the Northeast. The offense: the museum sought to “create an inclusive and impactful experience, which is aligned with DEI principles.” So too an academic study of how the plastics industry influenced municipal policy across the country. ChatGPT’s substantive analysis was limited to: “#DEI.”

Beyond this incoherence was something far more disturbing. In numerous cases, ChatGPT’s “DEI” classification was based solely on a grant description’s reference to a particular race or gender. For instance, it flagged funding for a documentary about the Colfax Massacre because the film “explores a historical event that significantly impacted Black civil rights.” Other termination rationales included, “This proposal aims to expand understanding of Black life and geography during slavery,” and, “The book explores a Black-led project.”

DOGE endorsed this crude approach. One of the DOGE members testified that funding for a documentary surfacing the slave labor of Jewish women during the Holocaust was properly canceled because it was “specifically focused on Jewish cultures” and the “voices of the females in that culture.” In the end, DOGE rubber-stamped ChatGPT’s outputs, adopting them wholesale as the basis for terminating more than 1,400 grants.

The court found the DOGE cuts unconstitutional multiple times over. Terminating funding based on its purported connection to “disfavored ideas” violated the First Amendment, and canceling grants because they made reference to a particular race, gender, or other protected class defied the Constitution’s promise of equal protection.

Thank you!

You have successfully joined our subscriber list.

DOGE’s use of ChatGPT was relevant to the court’s analysis. The bluntness—indeed, candor—of the dialogue between DOGE and ChatGPT made the government’s discriminatory motivation clear. The user explicitly asked the chatbot to discriminate; the chatbot obliged. Recall, for example, the grant canceled because ChatGPT noted that it “explore[d] a Black-led project.” This kind of smoking gun is extremely rare in constitutional litigation.

For its part, the government tried to distance itself from ChatGPT’s outputs. It insisted that the exchange with ChatGPT did not taint the ultimate termination decisions, and instead merely provided “context.” The court understood the government to be saying, “It wasn’t me; it was ChatGPT,” and rejected the defense. “Here,” the court found, “DOGE selected the AI tool, formulated the prompt, and defined the operative viewpoint-based criterion: whether a grant relates at all to DEI.” DOGE “adopted the classifications and supporting rationales that were generated by ChatGPT to justify terminating grant after grant,” and the court ruled that “those classifications are the Government’s own for constitutional purposes.”

This is the key insight of Learned Societies: LLMs are tools. Like any other tool, they can be used prudently or irresponsibly. To be sure, unlike, say, legal research databases, word processors, or pens, LLMs are generative—of prose, of answers to questions, of rationales. Consequently, agencies might be tempted, both in the funding and regulatory contexts, to treat LLMs as though they were themselves decision-makers. That could mean reflexively ratifying or rubber-stamping an LLM’s output, as DOGE did. It could also reflect a more profound category error that Learned Societies alludes to: treating an LLM as though it were itself a source of authority and expertise—an “independent, intervening actor,” in the court’s words.

Agencies that treat LLMs as decision-makers—either because of a genuine belief in the high-verisimilitude text they produce or a desire to “flood the zone” with hastily written rules—open themselves to significant risk. As was on vivid display in Learned Societies, LLMs have a penchant to say inapt and incorrect things and have proven ill-suited to push back on misguided or malign instructions. Outsourcing decision-making to tools with these tendencies creates self-evident issues. Agency regulations—which determine, among many other things, the integrity of the US financial system and the safety of its transportation networks—should be well-founded, considered, and reasonable. Over-reliance on LLMs also creates legal problems, as I have explained in Tech Policy Press before. Among other things, arbitrary and unsupported agency actions are unlawful.

This is not to say that regulators cannot responsibly and productively use LLMs. Indeed, LLMs might prove well-suited to many tasks agency policymakers must perform: for instance, reviewing relevant literatures, articulating competing views on a particular matter, identifying weaknesses in draft regulatory proposals, and drafting legal and policy analyses. But, like any other tool used in regulatory work, LLMs should be deployed in processes designed to take advantage of their strengths with safeguards keyed to their weaknesses. That means, as I have written about elsewhere, that agencies must train their staff on LLMs’ known limitations and develop protocols to mitigate LLMs’ tendencies toward sycophancy and hallucination. Most importantly, agencies should use LLMs only to the extent that accountable agency officials can meaningfully review and validate their work. An LLM might not be suitable for a task if it is impracticable for the agency to determine if the model has produced an accurate and reliable output. As agencies craft best practices for LLM use, the Learned Societies decision should serve as a handbook of what not to do.

LLMs are certain to have important applications in federal administration, but only with the right safeguards. Learned Societies is a wake-up call that plans to outsource regulatory decision-making to LLMs are misguided—and, if nothing else, will not work.

Support Tech Policy Press

If you've found our work helpful, consider supporting us.

Donate

Authors

Jordan Ascher

Jordan Ascher is Policy Counsel at Governing for Impact. He writes on a range of legal and governance topics, including the application of administrative law to federal agencies' use of artificial intelligence. He was previously a litigator in private practice and a law clerk to two federal circuit ...

Topics

Related

Perspective

DOGE’s Flops Shouldn’t Spell Doom for AI In GovernmentSeptember 5, 2025

Perspective

Using AI to Reform Government is Much Harder Than it LooksJune 3, 2025

DOGE Plan to Push AI Across the US Federal Government is Wildly DangerousMarch 6, 2025