Table of Contents
Early Years and Passion for AI
Ilya was born in the Soviet Union around 1985 or 1986. His exact birthday is not known. When he was 5 years old, his family moved to Israel. As a child, he would look at his hand and wonder'How can it be that this is my hand?'Ilya taught himself how to code when he was just seven, an early sign of his genius.
'I definitely was interested in AI from a fairly early age.'When he turned 16, his family decided to move again, and so when my family moved to Canada, I remember the first thing I did was go to the Toronto Public Library and try to find a book on machine learning. From that point, Ilya was hooked, and all he could think about was AI. He decided that his main goal in life is to build AGI.
Academic Pursuits and Encounter with Geoffrey Hinton
Ilya attended the University of Toronto, where he earned his bachelor's, master's, and PhD degrees. However, none of those degrees would impact his life as much as his next decision. This university had something that no other school could offer: a certain Professor Geoffrey Hinton, one of the greatest pioneers in the history of AI. Back then, most people saw AI as a total joke, but not Ilya. He desperately wanted to join Hinton's deep learning lab. He knocked on his door every day, hoping to be accepted. When Hinton asked him to make an appointment, Ilya instantly replied,'How about now?'It didn't take long before Hinton realized that Ilya was special. He had the ability to discover things that took others years to find.
Entrance into Hinton's Lab Amidst the AI Winter
Seeing the talent he was dealing with, Hinton accepted Ilya into his lab. But the timing couldn't be worse—the field was going through an AI winter, a period where interest in AI was at an all-time low. To add fuel to the fire, the AI community didn't really like Jeff Hinton. He kept pushing the idea of neural networks, but back then, they were completely useless. Everyone doubted Jeff and Ilya, which only made them more determined. Years went by with little to no progress. Computers simply weren't good enough at the time. But Jeff Hinton and his team kept on pushing. He knew that deep learning would eventually work.The Deep Learning Revolution: ImageNet Competition
In 2012, there came an opportunity to prove it—the ImageNet competition. This was a challenge to see who could make the best image recognition algorithm. Teams of researchers from all around the world competed for the number one spot. Luckily, Jeff and Ilya had an ace up their sleeve. Together with Alex Krizhevsky, they created AlexNet, which shocked the world.Deep Learning Revolution: ImageNet Triumph
Thanks to their idea to use deep neural networks and train them on GPUs, Ilya and the AlexNet team humiliated all competition. But more importantly, this single event showed everyone the immense potential of deep learning. This was the start of the deep learning revolution, and Ilya was at the center of it.'I just don't want to bet against deep learning. I want to make the biggest possible bet on deep learning.'
Google Days and Contributions
After the ImageNet competition, Ilya joined Jeff Hinton's new research company, DNN Research. However, just four months later, in March 2013, Google acquired the startup for an undisclosed amount. Of course, Google made sure to hire Ilya as a research scientist. While Ilya was at Google Brain, he worked on a bunch of interesting projects. He is one of the co-authors of the famous AlphaGo paper, among people like Demis Hassabis and David Silver. He also worked on TensorFlow, making it easier to use for researchers. However, Ilya's biggest contribution was the invention of the sequence-to-sequence learning algorithm. In fact, Ilya's new algorithm would eventually lead to the creation of the Transformer, which he would use to change the world. But more on that later.The Pivotal Decision to Join OpenAI
Google was on top of the AI world, and Ilya was one of its most valued researchers. But one day, he received a strange email.'Then actually, one day, I received an email from Sam saying, "Hey, let's hang out with some cool people."'Little did Ilya know that this invitation to hang out would completely change his life.
'We were having a dinner to discuss AI in the future and kind of just what might be possible and whether we could do something positive to affect it. So, my co-founders at OpenAI, that's Elon, Sam, Ilya, and other people, were all there.'This legendary dinner was where the original OpenAI vision first took shape. Everyone was excited and hopeful, including Ilya. However, there was a problem. Ilya was still working at Google with Demis Hassabis. Ilya went back and forth several times.
He would say he's going to join OpenAI, then Demis would convince him not to, then I would convince him to do so, and then this went back and forth several times. Ultimately, he decided to join OpenAI. If Ilya refused to join, if he stayed at Google, then OpenAI and ChatGPT would not exist today. Luckily for us, Ilya did end up joining as the research director.
Challenges of OpenAI's Nonprofit Beginnings
The early days of OpenAI were rough. The company was a nonprofit, so finding investors was a nightmare. It turns out that people don't like investing in something knowing that they will never see that money ever again. Fortunately, Elon Musk decided to commit $1 billion to the project. This allowed the team at OpenAI to hire some of the best people in the field, and with the influx of new talent came some amazing breakthroughs. In 2016, they released OpenAI Gym. In 2017, they released RoboSumo and Universe.Dota 2 Breakthrough and Financial Struggles
In 2018, OpenAI shocked the world with a team of Dota 2 bots that could play the game better than even professional players. These projects made OpenAI more and more famous. However, they also had a cost. The company was spending millions of dollars a month on cloud computing. What's worse, OpenAI was renting computing power from Google, their biggest competitor. Elon was starting to believe that OpenAI had fallen behind other players like DeepMind. He proposed to the board of directors with an idea: get rid of Sam Altman and make him the CEO of the company. However, the board rejected this idea, and so Elon Musk packed his bags and left OpenAI. Just like that, the startup lost its biggest investor. This created a massive problem. OpenAI was still a nonprofit and was spending millions of dollars to train its AI models. If they didn't find new investors fast, they would go bankrupt.A Lifesaving Partnership with Microsoft
Sam Altman tried to use his connections in the venture capital world to get people to invest in OpenAI, but every single person turned him down. The company needed way too much money for a nonprofit. Sam, Ilya, and Greg had to think quickly. They needed to come up with a solution. Otherwise, all of their progress and hard work would go to waste. If OpenAI went under, Google would become the winner of the AI race. If only there was a company that wanted Google to fail. Sam Altman decided to give Microsoft a call. Luckily, the CEO of Microsoft agreed to meet with him.Related Posts
Microsoft's Lifesaving Partnership
Everything depended on this meeting. If Sam couldn't convince Satya Nadella to invest, it would be the end of OpenAI. Sam Altman flew to Seattle and pulled off a miracle.'Today, we are very excited to announce a strategic partnership with OpenAI.'Microsoft agreed to invest $1 billion, but not only that, Microsoft also gave OpenAI access to Azure, their cloud computing platform. That way, OpenAI didn't have to pay Google anymore.
'And Microsoft has been a very, very good partner for us.'Microsoft provided OpenAI with a plethora of resources, which they immediately put to good use.
Evolution of OpenAI's GPT Models
In 2018, Ilya and other AI researchers at OpenAI created the original GPT model, also known as GPT-1. This was the first major use of the new Transformer architecture. A year later, in 2019, OpenAI made waves in the AI community by releasing GPT-2. This new model showed the incredible potential of large language models. But it also achieved another purpose, a purpose that was close to Ilya's heart: GPT-2 managed to impress Geoffrey Hinton himself.'GPT-2, which was one of the earlier language models, amazed me.'However, all of this success had a price. By partnering with Microsoft, OpenAI stopped being a nonprofit entity. This didn't sit well with the AI community. To make matters worse, the company was releasing less and less of their work open source. The final blow came in 2020 with the release of GPT-3. This marked the turning point when OpenAI stopped being OpenAI. But Ilya himself said that the move to close source was about competition, not safety.
All controversy aside, GPT-3 was pretty good. Scratch that, it was really good. However, it didn't really garner that much attention. Sure, everyone within the AI community knew about it, but the impact on the general public wasn't that great.
The Birth of ChatGPT and Ilya's Enduring Impact
Until Sam Altman got an idea. What if OpenAI gave GPT-3 a nice user-friendly interface? Maybe it just needed a clean UI that the average person could use. But the other OpenAI founders weren't so excited. They wanted to keep working on better and better models. Why bother spending time and resources on some user interface? However, Sam persisted. He believed in his idea so much that he ended up convincing Ilya and Greg. So, they took the GPT-3 model, improved it a bit with a process called RLf, and called the new model GPT-3.5. They decided to go with the simplest user interface possible, ChatGPT. Now they just needed a name. The most obvious choice was connecting the name of the UI with the name of the language model, and so ChatGPT was born. We all know what happened next.ChatGPT's Explosive Success
Sam Altman's intuition turned out to be right, and overnight the world was changed. One million users in 5 days, the fastest growing product in history. Everyone was shocked by the success of ChatGPT, including the man who built it.'A thing which has given me, me personally, an endless amount of joy is when my parents told me that their friends use ChatGPT in their daily lives.'But Ilya's impact on the world goes far beyond ChatGPT. In 2015, MIT named him one of the 35 most important innovators in the world. Time Magazine put him as one of the most influential people in AI. In total, Ilya has been cited over 427,000 times, making him one of the most cited computer scientists in history. For the last 11 years, he has been at the forefront of the AI revolution, and I would bet that 50 years from now, once AI really changes the world, Ilya will be remembered as one of the most important scientists of all time, up there with people like Isaac Newton, Albert Einstein, and Alan Turing.
'I tried really hard. I gave it everything I got, and that worked so far.'
Some questions
1. Who is Ilya Sutskever?
Ilya Sutskever is the Chief Scientist at OpenAI and a key figure in the field of artificial intelligence.
2. What sparked Ilya's interest in AI?
Ilya's interest in AI began at a young age when he taught himself to code at seven. His fascination grew, leading to a lifelong commitment to building Artificial General Intelligence (AGI).
3. What pivotal decision did Ilya make during his academic pursuits?
During his academic journey at the University of Toronto, Ilya made a significant decision to join Professor Geoffrey Hinton's deep learning lab, despite the AI field going through a challenging period known as the AI winter.
4. What breakthrough marked the start of the deep learning revolution?
The breakthrough came in 2012 with the ImageNet competition, where Ilya, along with Jeff Hinton and Alex Krizhevsky, created AlexNet, showcasing the potential of deep learning and leading to a revolution in the field.
5. What was Ilya's contribution during his time at Google Brain?
While at Google Brain, Ilya contributed to projects such as AlphaGo and TensorFlow. His notable achievement was the invention of the sequence-to-sequence learning algorithm, a precursor to the Transformer architecture.
6. Why did Ilya decide to join OpenAI?
Ilya joined OpenAI after receiving an invitation from Sam Altman during a dinner discussing the future of AI. Despite back-and-forth decisions, he became the research director, crucial for OpenAI's existence.
7. How did Elon Musk support OpenAI during its nonprofit phase?
Elon Musk committed $1 billion to OpenAI during its nonprofit phase, enabling the company to hire top talent and make breakthroughs, such as the release of OpenAI Gym and other projects.
8. What financial challenges did OpenAI face in 2018?
OpenAI faced financial challenges, with millions spent on cloud computing, mainly rented from Google. Elon Musk's proposed solution, involving changes in leadership, was rejected by the board, leading to Musk's departure.
9. How did OpenAI overcome financial struggles?
Sam Altman, Ilya, and Greg sought a solution to OpenAI's financial woes. Facing rejections from investors, Sam approached Microsoft, securing a $1 billion investment and access to Azure, Microsoft's cloud computing platform.
10. What marked the transition of OpenAI from nonprofit to a different entity?
In 2018, OpenAI, in partnership with Microsoft, transitioned from being a nonprofit entity. This move generated controversy within the AI community due to reduced open-source contributions.
11. What impact did GPT-2 have on the AI community?
GPT-2, released in 2019, impressed Geoffrey Hinton and showcased the potential of large language models, but OpenAI's shift to closed source with the release of GPT-3 in 2020 raised concerns about openness and competition.
12. What led to the creation of ChatGPT?
Sam Altman's idea to give GPT-3 a user-friendly interface led to the creation of ChatGPT. Despite initial skepticism from other founders, the model gained massive success, with one million users in just five days.
13. How did ChatGPT impact Ilya personally?
ChatGPT's success brought endless joy to Ilya, especially when he learned that his parents' friends used ChatGPT in their daily lives.
14. What recognition has Ilya received in the AI community?
In 2015, MIT named Ilya one of the 35 most important innovators globally. Time Magazine recognized him as one of the most influential people in AI. With over 427,000 citations, he is among the most cited computer scientists in history.
15. What is Ilya's perspective on the future of AI?
Ilya expressed a strong belief in the potential of deep learning, emphasizing the importance of making the biggest possible bet on its success.
16. What role did Ilya play in OpenAI's Dota 2 breakthrough?
In 2018, Ilya was part of the team that shocked the world with Dota 2 bots outperforming professional players, showcasing OpenAI's capabilities but also contributing to financial challenges.
17. How did Microsoft's partnership with OpenAI contribute to the company's progress?
Microsoft's $1 billion investment and access to Azure provided OpenAI with crucial resources, alleviating financial burdens and enabling the company to continue its work.
18. How did Ilya's early interest in AI evolve during his time at the University of Toronto?
During his time at the University of Toronto, Ilya's interest in AI deepened, especially after joining Geoffrey Hinton's lab and contributing to the field despite the challenges of the AI winter.
19. What impact did Ilya and the AlexNet team have on the ImageNet competition?
The AlexNet team's use of deep neural networks and GPU training led to a decisive victory in the ImageNet competition in 2012, marking the beginning of the deep learning revolution with Ilya at its center.
20. How did Ilya's decision to join OpenAI impact the company's existence?
Ilya's decision to join OpenAI, despite uncertainties, as the research director played a pivotal role in the formation and success of OpenAI and projects like ChatGPT.
21. What challenges did OpenAI face as a nonprofit organization?
OpenAI encountered challenges in finding investors as a nonprofit organization, prompting Elon Musk's $1 billion commitment to the project, which proved crucial for OpenAI's progress.
22. How did Ilya's academic pursuits lead him to Geoffrey Hinton's lab?
Ilya's determination and talent led him to Geoffrey Hinton's lab at the University of Toronto, where he contributed to groundbreaking work in deep learning despite the prevailing skepticism in the AI community.
23. What motivated Ilya to pursue AGI as his main life goal?
Ilya's fascination with AI, starting from a young age, culminated in his decision to pursue AGI as his main life goal, a commitment that has driven his contributions to the field.
24. What is Ilya's perspective on betting against deep learning?
Ilya expressed a strong stance against betting against deep learning, emphasizing his desire to make the biggest possible bet on its success and impact on the future.