San Francisco orders Apple, Google to remove nudify apps from app stores

Apple did not respond to Ars’ request to comment.

The Grok elephant in the room

Although Apple has removed some apps flagged by researchers, the iPhone maker otherwise has remained notably silent amid calls to police nudification apps more aggressively in its App Store.

App stores’ detection abilities matter, as Wired’s report noted that researchers have recently warned that some harmful apps have gotten better at avoiding app store removals by only promoting face-swapping features while hiding nudifying capabilities. In a May preprint paper, researchers identified 420 apps touted as generic face-swapping tools and tested 155 to see if they could be used to sexualize images. In 70 percent of apps tested, nudification was possible.

There’s also the Grok elephant in the room to consider. This week, xAI filed a lawsuit confirming that it found instances of Grok-generated child sexual abuse materials (CSAM) and other nonconsensual intimate imagery (NCII) targeting adults. To prevent the Grok misuse, xAI sued the user allegedly responsible for prompting Grok to generate the illegal content, but the question remains whether xAI is liable for the outputs.

Ever since the Grok scandal erupted, app stores have faced questions on whether xAI’s chatbot is violating app store policies against harmful content. Back in April, Apple told Senators that it had privately threatened to remove Grok, NBC News reported, but the xAI app remains in the app store today.

So long as it’s possible to use Grok to generate illegal content, app stores will likely continue to face pressure to take action to prevent harmful outputs, just as Chiu said that app stores should be doing with all apps allowing nudification if they want to comply with California deepfake laws.

Although Google’s spokesperson said that Google continuously tests generative AI apps to make sure they have safeguards preventing NCII and CSAM creation, Jackson did not respond to Ars’ request to comment on whether the way that Grok performs today is an exception to Google’s policy.

In his letters to app stores, Chiu did not request Grok removals, but it is clear he is hoping that his demands will help broadly tighten up enforcement in both app stores, starting with removing the apps that his office flagged.

“My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future,” he says. “It’s our hope that these companies will do the right thing—but if they don’t, we will have to consider all of our legal options.”