Мы в Telegram
Добавить новость
ru24.net
Vox
Май
2024

“I lost trust”: Why the OpenAI team in charge of safeguarding humanity imploded

0
Vox 

Sam Altman is seen in profile against a dark background with a bright light overhead.
Sam Altman is the CEO of ChatGPT maker OpenAI, which has been losing its most safety-focused researchers. | Joel Saget/AFP via Getty Images

Company insiders explain why safety-conscious employees are leaving.

For months, OpenAI has been losing employees who care deeply about making sure AI is safe. Now, the company is positively hemorrhaging them.

Ilya Sutskever and Jan Leike announced their departures from OpenAI, the maker of ChatGPT, on Tuesday. They were the leaders of the company’s superalignment team — the team tasked with ensuring that AI stays aligned with the goals of its makers, rather than acting unpredictably and harming humanity.

They’re not the only ones who’ve left. Since last November — when OpenAI’s board tried to fire CEO Sam Altman only to see him quickly claw his way back to power — at least five more of the company’s most safety-conscious employees have either quit or been pushed out.

What’s going on here?

If you’ve been following the saga on social media, you might think OpenAI secretly made a huge technological breakthrough. The meme “What did Ilya see?” speculates that Sutskever, the former chief scientist, left because he saw something horrifying, like an AI system that could destroy humanity.

But the real answer may have less to do with pessimism about technology and more to do with pessimism about humans — and one human in particular: Altman. According to sources familiar with the company, safety-minded employees have lost faith in him.

“It’s a process of trust collapsing bit by bit, like dominoes falling one by one,” a person with inside knowledge of the company told me, speaking on condition of anonymity.

Not many employees are willing to speak about this publicly. That’s partly because OpenAI is known for getting its workers to sign offboarding agreements with non-disparagement provisions upon leaving. If you refuse to sign one, you give up your equity in the company, which means you potentially lose out on millions of dollars.

One former employee, however, refused to sign the offboarding agreement so that he would be free to criticize the company. Daniel Kokotajlo, who joined OpenAI in 2022 with hopes of steering it toward safe deployment of AI, worked on the governance team — until he quit last month.

“OpenAI is training ever-more-powerful AI systems with the goal of eventually surpassing human intelligence across the board. This could be the best thing that has ever happened to humanity, but it could also be the worst if we don’t proceed with care,” Kokotajlo told me this week.

OpenAI says it wants to build artificial general intelligence (AGI), a hypothetical system that can perform at human or superhuman levels across many domains.

“I joined with substantial hope that OpenAI would rise to the occasion and behave more responsibly as they got closer to achieving AGI. It slowly became clear to many of us that this would not happen,” Kokotajlo told me. “I gradually lost trust in OpenAI leadership and their ability to responsibly handle AGI, so I quit.”

And Leike, explaining in a thread on X why he quit as co-leader of the superalignment team, painted a very similar picture Friday. “I have been disagreeing with OpenAI leadership about the company’s core priorities for quite some time, until we finally reached a breaking point,” he wrote.

OpenAI did not respond to a request for comment in time for publication.

Why OpenAI’s safety team grew to distrust Sam Altman

To get a handle on what happened, we need to rewind to last November. That’s when Sutskever, working together with the OpenAI board, tried to fire Altman. The board said Altman was “not consistently candid in his communications.” Translation: We don’t trust him.

The ouster failed spectacularly. Altman and his ally, company president Greg Brockman, threatened to take OpenAI’s top talent to Microsoft — effectively destroying OpenAI — unless Altman was reinstated. Faced with that threat, the board gave in. Altman came back more powerful than ever, with new, more supportive board members and a freer hand to run the company.

When you shoot at the king and miss, things tend to get awkward.

Publicly, Sutskever and Altman gave the appearance of a continuing friendship. And when Sutskever announced his departure this week, he said he was heading off to pursue “a project that is very personally meaningful to me.” Altman posted on X two minutes later, saying that “this is very sad to me; Ilya is … a dear friend.”

Yet Sutskever has not been seen at the OpenAI office in about six months — ever since the attempted coup. He has been remotely co-leading the superalignment team, tasked with making sure a future AGI would be aligned with the goals of humanity rather than going rogue. It’s a nice enough ambition, but one that’s divorced from the daily operations of the company, which has been racing to commercialize products under Altman’s leadership. And then there was this tweet, posted shortly after Altman’s reinstatement and quickly deleted:

So, despite the public-facing camaraderie, there’s reason to be skeptical that Sutskever and Altman were friends after the former attempted to oust the latter.

And Altman’s reaction to being fired had revealed something about his character: His threat to hollow out OpenAI unless the board rehired him, and his insistence on stacking the board with new members skewed in his favor, showed a determination to hold onto power and avoid future checks on it. Former colleagues and employees came forward to describe him as a manipulator who speaks out of both sides of his mouth — someone who claims, for instance, that he wants to prioritize safety, but contradicts that in his behaviors.

For example, Altman was fundraising with autocratic regimes like Saudi Arabia so he could spin up a new AI chip-making company, which would give him a huge supply of the coveted resources needed to build cutting-edge AI. That was alarming to safety-minded employees. If Altman truly cared about building and deploying AI in the safest way possible, why did he seem to be in a mad dash to accumulate as many chips as possible, which would only accelerate the technology? For that matter, why was he taking the safety risk of working with regimes that might use AI to supercharge digital surveillance or human rights abuses?

For employees, all this led to a gradual “loss of belief that when OpenAI says it’s going to do something or says that it values something, that that is actually true,” a source with inside knowledge of the company told me.

That gradual process crescendoed this week.

The superalignment team’s co-leader, Jan Leike, did not bother to play nice. “I resigned,” he posted on X, mere hours after Sutskever announced his departure. No warm goodbyes. No vote of confidence in the company’s leadership.

Other safety-minded former employees quote-tweeted Leike’s blunt resignation, appending heart emojis. One of them was Leopold Aschenbrenner, a Sutskever ally and superalignment team member who was fired from OpenAI last month. Media reports noted that he and Pavel Izmailov, another researcher on the same team, were allegedly fired for leaking information. But OpenAI has offered no evidence of a leak. And given the strict confidentiality agreement everyone signs when they first join OpenAI, it would be easy for Altman — a deeply networked Silicon Valley veteran who is an expert at working the press — to portray sharing even the most innocuous of information as “leaking,” if he was keen to get rid of Sutskever’s allies.

The same month that Aschenbrenner and Izmailov were forced out, another safety researcher, Cullen O’Keefe, also departed the company.

And two weeks ago, yet another safety researcher, William Saunders, wrote a cryptic post on the EA Forum, an online gathering place for members of the effective altruism movement, who have been heavily involved in the cause of AI safety. Saunders summarized the work he’s done at OpenAI as part of the superalignment team. Then he wrote: “I resigned from OpenAI on February 15, 2024.” A commenter asked the obvious question: Why was Saunders posting this?

“No comment,” Saunders replied. Commenters concluded that he is probably bound by a non-disparagement agreement.

Putting all of this together with my conversations with company insiders, what we get is a picture of at least seven people who tried to push OpenAI to greater safety from within, but ultimately lost so much faith in its charismatic leader that their position became untenable.

“I think a lot of people in the company who take safety and social impact seriously think of it as an open question: is working for a company like OpenAI a good thing to do?” said the person with inside knowledge of the company. “And the answer is only ‘yes’ to the extent that OpenAI is really going to be thoughtful and responsible about what it’s doing.”

With the safety team gutted, who will make sure OpenAI’s work is safe?

With Leike no longer there to run the superalignment team, OpenAI has replaced him with company co-founder John Schulman.

But the team has been hollowed out. And Schulman already has his hands full with his preexisting full-time job ensuring the safety of OpenAI’s current products. How much serious, forward-looking safety work can we hope for at OpenAI going forward?

Probably not much.

“The whole point of setting up the superalignment team was that there’s actually different kinds of safety issues that arise if the company is successful in building AGI,” the person with inside knowledge told me. “So, this was a dedicated investment in that future.”

Even when the team was functioning at full capacity, that “dedicated investment” was home to a tiny fraction of OpenAI’s researchers and was promised only 20 percent of its computing power — perhaps the most important resource at an AI company. Now, that computing power may be siphoned off to other OpenAI teams, and it’s unclear if there’ll be much focus on avoiding catastrophic risk from future AI models.

To be clear, this does not mean the products OpenAI is releasing now — like the new version of ChatGPT, dubbed GPT-4o, which can have a natural-sounding dialogue with users — are going to destroy humanity. But what’s coming down the pike?

“It’s important to distinguish between ‘Are they currently building and deploying AI systems that are unsafe?’ versus ‘Are they on track to build and deploy AGI or superintelligence safely?’” the source with inside knowledge said. “I think the answer to the second question is no.”

Leike expressed that same concern in his Friday thread on X. He noted that his team had been struggling to get enough computing power to do its work and generally “sailing against the wind.”

Most strikingly, Leike said, “I believe much more of our bandwidth should be spent getting ready for the next generations of models, on security, monitoring, preparedness, safety, adversarial robustness, (super)alignment, confidentiality, societal impact, and related topics. These problems are quite hard to get right, and I am concerned we aren’t on a trajectory to get there.”

When one of the world’s leading minds in AI safety says the world’s leading AI company isn’t on the right trajectory, we all have reason to be concerned.




Moscow.media
Частные объявления сегодня





Rss.plus



Ежегодная церемония вручения Всероссийской премии Евгения Зубова прошла в библиотеке Видного

США и Европа "подготавливают" Россию и Ближний Восток для переселений?!

В Подмосковье сотрудники Росгвардии спасли пожилого мужчину, который оказался один дома и плохо себя почувствовал

В ЖК «Восточное Бутово» продолжают строительство шестого по счету детского сада


Об отношении к русским в Азербайджане

Уникальный интерактивный проект Ставропольской психиатрической больницы на выставке-форуме «Россия»

CЛД «Печора» филиала «Северный» ООО «ЛокоТех-Сервис» получило сертификацию для технического обслуживания локомотивов серии 2ТЭ25КМ в объёме ТО-2.

Новые электрические щетки Revyline RL 085 Black доступны в филиале марки в Ставрополе


Four-time Man Utd trophy winner, 37, looks unrecognisable as he heads to training with dramatic new look

Watch awkward moment baffled French Open teen is ‘teleported’ into Eurosport studio for live TV interview

Andy Murray cracks brutal joke as Brit star’s Wimbledon preparations are disrupted by bizarre injury 

Big 12 commissioner Brett Yormark addresses Arizona Board of Regents, asks for continued investment in athletics


Торговые настольные электронные весы CAS PR-15P

SURREAL NAMIBIA

Бурятский Театр кукол Ульгэр в Улан-Улан-Удэ показал в Этнографическом музее народов Забайкалья перед детьми и родителями в семейный праздник сказку Колобок - Новости и Культура, Дети и Россия

Трасса М-12 Восток будет продлена до Тюмени в 2025 году


I tried to recreate Marvel's Iron Man in this movie studio sim, and my version was so bad my own father stormed out of the theater

Мафия-НН: Все в сборе, правила с тревогой выслушаны, роли прочитаны, и сразу же в бой!

How to watch the 2024 PC Gaming Show: our 10th annual summer showcase

Показ геймплея тактики SteamWorld Heist II



США и Европа "подготавливают" Россию и Ближний Восток для переселений?!

СОТРУДНИКИ СОБР «СТОЛИЦА» СТАЛИ ИНСТРУКТОРАМИ НА СБОРАХ ПО БЕСПАРАШЮТНОМУ ДЕСАНТИРОВАНИЮ СПЕЦНАЗОВЦЕВ РОСГВАРДИИ В ВОРОНЕЖСКОЙ ОБЛАСТИ

В ПРЕДДВЕРИИ ДНЯ РОССИИ РОСГВАРДЕЙЦЫ ПРОВЕЛИ ПРАЗДНИЧНОЕ МЕРОПРИЯТИЕ ДЛЯ ДЕТЕЙ

РОСГВАРДЕЙЦЫ ЗАДЕРЖАЛИ МУЖЧИНУ С ПОДДЕЛЬНЫМИ ДОКУМЕНТАМИ В МОСКВЕ




«Типичный ребенок»: панда Катюша устроила игры на качелях

В работе мессенджера Telegram произошел сбой

СОТРУДНИКИ СОБР «СТОЛИЦА» СТАЛИ ИНСТРУКТОРАМИ НА СБОРАХ ПО БЕСПАРАШЮТНОМУ ДЕСАНТИРОВАНИЮ СПЕЦНАЗОВЦЕВ РОСГВАРДИИ В ВОРОНЕЖСКОЙ ОБЛАСТИ

На Ближнем Востоке зарождается союз, который перевернет мир. Выгоды для России


Корь в России: от вспышки к стабилизации

Цивилёва: хирург закрыл собой вход в палатку с семью ранеными бойцами СВО

В Подмосковье арестован стрелявший в полицейского велосипедист

Водитель фургона сдавал назад и наехал на мужчину на западе Москвы


Джокович снялся с Открытого чемпионата Франции по теннису

«Сейчас даже счёт не помню»: Андреева сенсационно победила Соболенко и вышла в полуфинал «Ролан Гаррос»

Экс-теннисист Ольховский: Рублев не успел набрать форму на "Ролан Гаррос"

Теннисистка Андреева заявила, что хочет войти в топ-20 рейтинга WTA ради собаки


«Субтропическая погода»: москвичей предупредили о жаре и грозах на следующей неделе

Посол в Израиле Викторов: Москва разделяет радость близких освобожденных пленных

"Голяк": захватывающий сериал, который завоевал миллионы сердец по всему миру!

Пропавшую многодетную мать нашли в брюхе питона в Индонезии


Музыкальные новости

Приглашенные солистки выступят в опере «Руслан и Людмила»

Новости и Культура, Дети и Россия: кукольный Театр Ульгэр выступил в Улан-Улан-Удэ в Этнографическом музее народов Забайкалья перед детьми и родителями в семейный праздник

Бурятский Театр кукол Ульгэр в Улан-Улан-Удэ показал в Этнографическом музее народов Забайкалья перед детьми и родителями в семейный праздник сказку Колобок - Новости и Культура, Дети и Россия

А что, так можно было? "Голый концерт" Бузовой поставил ребром вопрос о цензуре в России



СОТРУДНИКИ СОБР «СТОЛИЦА» СТАЛИ ИНСТРУКТОРАМИ НА СБОРАХ ПО БЕСПАРАШЮТНОМУ ДЕСАНТИРОВАНИЮ СПЕЦНАЗОВЦЕВ РОСГВАРДИИ В ВОРОНЕЖСКОЙ ОБЛАСТИ

США и Европа "подготавливают" Россию и Ближний Восток для переселений?!

РОСГВАРДЕЙЦЫ ЗАДЕРЖАЛИ МУЖЧИНУ С ПОДДЕЛЬНЫМИ ДОКУМЕНТАМИ В МОСКВЕ

США и Европа "подготавливают" Россию и Ближний Восток для переселений?!


«Солнечная Азбука» в особом семейном центре «Семь-Я»

США и Европа "подготавливают" Россию и Ближний Восток для переселений?!

На пленарном заседании ПМЭФ-2024 Владимир Путин дал оценку состояния российской экономики и обозначил ориентиры на будущее

Медиа съезжают последними // «Газпром» завершает перевод структур из Москвы в Санкт-Петербург


Асфальтирование дворов услуги в Петербурге

В Москве 14-летний мальчик ранил ножом 16-летнего юношу на футбольном поле

Асфальтирование площадок услуги в Петербурге

Асфальтирование пешеходных дорожек Санкт-Петербург


«Всуе не упоминать!»: Путин на ПМЭФ остудил ядерные фантазии и вспомнил Сталина

Профессор Кошкин: Запад в шоке от военных кораблей России в Карибском море

Путин: Москва оставляет за собой право поставок вооружения партнерам

Путин пригрозил оружием Западу, Киев закрыл поездки по Европе, США ввели санкции против Грузии – большие итоги недели от РИА «Новый День»


Вирус эволюционирует: почему рано называть COVID-19 сезонным заболеванием



Гражданку Кыргызстана арестовали по обвинению в угрозах судье и прокурору, отправившим за решетку Навального


Мурашко: пятеро пострадавших при теракте в "Крокусе" остаются в больницах

Не просто кашель курильщика: чем опасна хроническая обструктивная болезнь легких

Вирус эволюционирует: почему рано называть COVID-19 сезонным заболеванием

Аллерголог Зайцева рассказала, как проявляется аллергия на солнечный свет


AFP: Байден объявил о выделении Киеву помощи на 225 млн долларов


Спортивные тренировки и мастер-классы пройдут на летнем фестивале в Москве

Мастер-классы по созданию крафтовых украшений проходят в центре Москвы

В Москве 14-летний мальчик ранил ножом 16-летнего юношу на футбольном поле

В Москве прошёл спортивный праздник для росгвардейцев




Собянин: В Москве ведется активная замена инженерных сетей в рамках реновации

Собянин: Москва — мировой лидер по динамике развития транспортной инфраструктуры

Собянин: Благодаря модернизации театры Москвы создают уникальные декорации

Собянин: Москва подписала ряд крупных соглашений на Петербургском форуме


На ПМЭФ-2024 Краснодарский край подписал 40 протоколов на 272 млрд рублей

Суд отправил под арест подозреваемого в убийстве пенсионера в парке Москвы

Байрактар: Турция и Россия определили дальнейшие шаги по проекту "газового хаба"

В Москве проходит акция по безопасному использованию СИМ


Пропавшую многодетную мать нашли в брюхе питона в Индонезии

«Субтропическая погода»: москвичей предупредили о жаре и грозах на следующей неделе

В Подмосковье арестован стрелявший в полицейского велосипедист

"Голяк": захватывающий сериал, который завоевал миллионы сердец по всему миру!


Терминал сбора данных (ТСД) промышленного класса SAOTRON RT42G

Россия, Новости, Культура, Театр, Дети: Радость, которую приносит Госцирк Бурятии

В Пулково задерживаются шесть рейсов

Минфин Архангельской области оценил муниципалитеты по уровню качества организации и осуществления бюджетного процесса


Глава Росавиации рассказал о перспективах открытия аэропорта Краснодара

Литературно-краеведческий коллаж «Мир добрых сказок и стихотворений – все это Пушкин, наш великий гений!»

На этой неделе загрязнение воздуха отмечали в четырех городах Крыма

Пьяный мужчина упал с верхней полки на девочку в поезде Москва - Симферополь


Полпредство Татарстана посетили коллеги из Алтая для изучения опыта работы

Взрыв произошел в пятиэтажке на севере Москвы

«Субтропическая погода»: москвичей предупредили о жаре и грозах на следующей неделе

Депутат парламента Гамбурга сбежала из Германии в Россию












Спорт в России и мире

Новости спорта


Новости тенниса
Ролан Гаррос

Семнадцатилетняя Андреева впервые сыграет в полуфинале турнира Большого шлема






Замминистра Черепнин: продукция из Подмосковья востребована в других странах

Культуру и заповедные территории 20 регионов России представят в Пекине

Медведев назвал пособником террористов потребовавшую его ареста организацию

Ликсутов: На ночных маршрутах совершают более 3,5 тыс. поездок