AI companies are in a state of anxiety as California tries to both develop and regulate the tech

• California is leading the world in the AI arms race.
• However, a new California law could change the course of the AI revolution forever.
• Tech giants like Meta and OpenAI argue the bill would stifle AI innovation.

Silicon Valley is divided over a landmark bill in California that could radically change the pace of AI development everywhere.

California Sen. Scott Wiener introduced SB 1047 in February, and it has since won support from legislators on both sides of the aisle.

It passed the Senate's Committee on Privacy and Consumer Protection in June and the Assembly's Appropriations Committee last week. The full Assembly is scheduled to vote on the bill by the end of the month.

California has established itself as a leader in the global AI arms race, with 35 of the top 50 AI companies based in the state, according to a report from the Brookings Institute.

California Gov. Gavin Newsom has worked to help further his state's status as a global AI pioneer. Earlier this year, California revealed a state worker training program, held a generative AI summit, and launched pilot projects to understand how the technology can address challenges like traffic congestion and language accessibility.

This month, the state unveiled a partnership with chipmaker Nvidia that will train the state's residents on how to use the technology to support job creation, promote innovation, and leverage AI to solve everyday problems.

But amid all the excitement around AI's potential, there is also no small amount of concern about its danger to humanity. That means California must walk a fine line between regulating the AI industry and stifling the growth it hopes to see.

At an AI event in San Francisco in May, Newsom said, "If we over-regulate, if we overindulge, if we chase the shiny object, we could put ourselves in a perilous position."

Newsom signed an executive order in September that called for several provisions to ensure the responsibe deployment of the technology and directed state agencies to examine its best uses.

Newsom has not publicly weighed in on SB 1047. The governor's office did not respond to a request for comment from Business Insider. If the bill is signed into law, though, it would be the most exhaustive attempt to regulate the AI industry yet.

What SB 1047 would change

The bill's authors say its goal is to "ensure the safe development of large-scale artificial intelligence systems by establishing clear, predictable, common-sense safety standards for developers of the largest and most powerful AI systems."

It would apply to companies developing models that cost over $100 million to train or use high levels of computing power. These companies would be required to safety test new technologies before making them publicly available. They would also have to build "full shutdown" capabilities into their models and be held liable for all technology applications.

The bill also establishes legal standards for AI developers — outlining "unlawful acts" for which California's attorney general could sue companies, setting protections for whistleblowers, and creating a board to set compute limits on AI models and issue regulatory guidance.

What Big Tech thinks

The developers the bill will most likely impact haven't been welcoming.

Companies like Meta, OpenAI, and Anthropic — which are funneling millions into building and training large language models — have lobbied state legislators for changes. Meta said the initial legislation would hamper innovation and make it less likely that companies would allow their model to be open source.

"The bill also actively discourages the release of open-source AI because providers would have no way to open-source models without facing untenable legal liability," Meta wrote in a letter to Wiener in June. That's likely to have ramifications on the small business ecosystem by lessening the likelihood of them using "free, readily available and fine-tuned models to create new jobs, businesses, tools, and services that often go on to be used by other businesses, governments, and civil society groups."

Anthropic, which has positioned itself as a safety-conscious AI company, wasn't happy with early versions of the bill and lobbied lawmakers for changes. It calls for a greater focus on deterring companies from building unsafe models rather than enforcing stringent laws before catastrophic incidents happen. It also suggests that companies that meet the $100 million threshold should be able to set their own standards for safety testing.

The bill has also drawn criticism from venture capitalists, executives, and other tech industry members. Anjney Midha, a general partner at Andreessen Horowitz, called it "one of the most anti-competitive proposals I've seen in a long time." He believes legislators should focus on "regulating specific high-risk applications and malicious end users."

California legislators made a handful of the suggested changes — all of which made it into the latest version of the bill. The updated version now prevents the California Attorney General's Office from suing AI developers before a catastrophic event happens. And while the bill initially called for a new government agency to oversee the enforcement of new laws, it has since pared that down to a board within the state's Government Operations Agency.

A spokesperson for Anthropic told BI that it's "reviewing the new bill language as it becomes available." Meta and OpenAI did not respond to a request for comment.

Smaller founders are apprehensive but also a bit more optimistic.

Arun Subramaniyam, the founder and CEO of enterprise-focused generative AI company Articul8, told BI that no one understands the new board's "sweeping powers" or how its directors will be appointed. He also believes the bill will impact companies beyond the Big Tech players since even well-funded startups might meet the $100 million training threshold.

At the same time, he said he supports the bill's establishment of a public cloud computing cluster, CalCompute, dedicated to researching the safe deployment of large-scale AI models. This cluster could bring more equity to researchers and groups that don't have the funds to evaluate AI models on their own. He also believes the reporting requirements on developers will increase transparency, benefiting Articul8's work in the long run.

He said the industry's future hinges on how general-purpose models are regulated. "It's good that the state is looking at regulating this early, but the vocabulary needs to be a little more balanced," he said.