Should AI companies become profit-maximizing firms?
Could this make our pro-innovation decisions more explicit?
Today I was fortunate to hear from a law student who works on making AI companies internalize negative externalities.
At first, I thought he meant externalities from existential risk, and was taken aback by the scale of these. I asked how such off-the-scale / naïvely-incommensurable costs could be internalized.
He started to reply:
Well, companies should be able to use the surplus from profits (which ideally reflect value provided to society) to pay for insurance (that provides coverage for costs they might impose on society).
I asked what % of their budget AI companies might need to spend on insurance. Couldn’t costs spiral out of control quickly for small companies developing general-purpose models?
If they have to spend so much on insurance, mightn’t this preclude altruistic endeavors, e.g. giving students free access (as Google does)?
Why do you think they give students free access? Is it altruistic?
(Gosh, I’m naïve and San Franciscan)—
It is not! They are profit-maximizing firms.
Are they?
First, I’m not sure that they literally are. It seems PBCs like OpenAI and Anthropic are at most mission-constrained profit-maximizers.
Then there’s the more interesting question—should they be?
I started to distinguish between two schools of thought:
Treat AI developers as profit-maximizing firms, force them to internalize externalities through liability regimes, use government subsidies to support innovation, and let markets take care of everything.
Treat AI developers as public-interest entities. Don’t impose expensive liability regimes: the AI companies can’t afford it, because they aren’t capturing all the surplus they provide (and nor should they).
Steelmaning treating AI companies as profit-maximizers
Say we’re considering requiring AI companies to buy expensive insurance. Treating AI companies as profit-maximizers—responsible for capturing all surplus they provide to society—favors it: the company can take care of itself (and shouldn’t be around if it can’t). Treating AI companies as public-interest entities introduces ambiguity: the company can plead mission to dodge liability.
Treating AI companies as profit-maximizers allows us to explicitly quantify the size of the liability and the size of the subsidy—rather than implicitly subsidizing risk (by avoiding too-onerous liability regimes).
I am contra paperwork but drawn to making things more explicit.
Is it realistic?
I think, if I were a decision-maker at an AI company, it would be very hard for me to act in consistently profit-maximizing ways. But the law student I talked to thought maybe this is unusual. Is it?
I’d like to say typical economic literature applies less because people aren’t so easily bought and sold in this market. You can’t build superintelligence without the right research leads; there are <100 sufficiently skilled research leads, and many of them are driven by non-financial motivations: they won’t just go to the firm that pays highest. Hence we cannot explicitly or implicitly (through imposing costs that price out public benefit entities) model AI developers as profit-maximizers and call it a day.
Which considerations seem salient to you?
I think the profit-maximization question is interesting but wouldn't use it as the metric for deciding liability. When a negative externality risk exists, someone ends up on the hook, either the company or the public. And I disagree that any current major lab qualifies as a "public-interest entity."
If we were comparing a true non-profit (like a less messy version of OpenAI's original charter) with a profit-maximizer, I'd be more open to considering liability shields. But PBCs are legally required to "balance" investor(profit maxer) financial interests with their stated public benefits. Given their massive ongoing capex requirements, labs have every incentive to keep investors happy. The structural pressures skew towards profit maximization and even given the benefit of the doubt, PBCs were never meant to be used to lessen corporate liability costs.
I'm also unconvinced that rate-limited, soon to be ad filled, free tiers LLMs represent genuine altruism. Adobe gave me free Photoshop in undergrad, and here I am still paying monthly 20 years later.
Currently, the point is moot because even the largest insurers want nothing to do with this market and couldn't "afford to pay is if an AI provider makes a mistake that ends up as . . . a systemic, correlated, aggregated risk.” https://archive.ph/TPz5r#selection-2267.34-2267.152
I do think there are some interesting ideas here though about building up a market: https://underwriting-superintelligence.com/
Overall, lots of interesting questions to think about! Thanks for the write-up!
This was fun, but the Delmore Effect strikes again. I could rename my blog to The Delmore Effect. I must avoid economists, because they snipe me with things I have decided not to spend my life on.