Putting skills to the test in the midst of changing technology

Akiba Takuya immersed himself in research on algorithms as an undergraduate student at the Department of Information Science and as a graduate student at the Graduate School of Information Science and Technology. Even so, he had two major shifts in his research career.

Shocking defeat to messy code

Akiba has been interested in computers since his childhood. He started programming in his first year of junior high school and later came across competitive programming through extracurricular activities. He continued to write programs throughout junior high and high school until he felt confident that his programming skills were second to none.

However, he suffered a shocking defeat that shattered his pride in a programming contest for high school students. The student who beat him, a mathematical genius, was not that knowledgeable about computers and programming, and his code-writing style was not sophisticated. Nevertheless, his messy code outperformed Akiba's clean code.

“The difference in performance was due to the algorithms used. The program is just a way to communicate with the computer. The algorithm, however, specifies how the computer does its job. What algorithm is used makes a big difference in efficiency. It was a shocking realization.”

Even the most well-written program will lose if the algorithm used is inadequate. This made Akiba realize the scientific depth behind programming and he became interested in learning about algorithms. During the liberal arts curriculum of his first two years in college, he devoted himself more and more to competitive programming.

“I was living for competitive programming. I did not want to do anything else, I even skipped classes," he recalls. This changed drastically when the second two years of the undergraduate program began.

“The curriculum in the Department of Information Science was excellent. It was everything that I wanted to learn,” Akiba says cheerfully.“The curriculum allows students to learn about all layers of computers and computation, from the most abstract, such as what computation is, to the most pragmatic, such as how CPUs are made. I have been making good use of the lessons I learned at that time in my work ever since,” he adds.

Creating a unique program

“I want to create programs better than those written by others, not just programs that anyone can create. My interest in algorithms is a means to achieve this," says Akiba. His graduate school research theme was algorithms for processing large amounts of data in real-world graph structures, such as connections on social media.

Although algorithm researchers are usually eager to ascertain what makes a calculation fast, it is difficult to express mathematically real-world graphical data. Thus, demonstrating that an algorithm is indeed the fastest or revealing the reason why a calculation is fast is challenging.

“Generally, running speed is the priority, and the reasons why a certain algorithm runs fast way are secondary. The program that runs the fastest wins.” His graduate research field, which is slightly different from general algorithmic research, was a good match for Akiba's ambition to, first and foremost, create a program unique to him.

After receiving his degree, he continued his research on algorithms as an assistant professor at the National Institute of Informatics. However, his research interest shifted to deep learning after about a year. “I loved solving hard problems," he says, "but the important problems were gradually being solved, and it wCreating a unique programas time to think about new topics to tackle. Deep learning was starting to produce results, and I thought it was a good time to possibly make a big impact."

Everything is written in the source code

Simultaneously changing his area of research and taking up a position at Preferred Networks, Akiba had to learn deep learning from scratch. Yet, he says he was not worried about catching up with those ahead of him. In addition to reading textbooks and papers, he highlights the importance of reading source code.

“The source code is the blueprint where everything is written,” he says. “Computers cannot read between the lines, so there are no such spaces in the source code either. By reading and executing the source code, you can reach a deeper understanding."

He also points out that in the case of novel and innovative technologies, everyone ends up starting almost from scratch. When competing at the cutting edge of constantly changing technologies, it is the fundamentals that are important, the universal principles of computers and computation he learned about in the Department of Information Science.

Competing with techincal prowess in the field of generative AI

After seven years of deep learning research, Akiba was ready to move on to the next stage of his career when the wave of generative AI arrived. He wanted to be at the center of this revolutionary technology and devote 100% of his attention to generative AI. So, he joined Sakana AI. Although he had zero experience with Large Language Models (LLMs) such as ChatGPT, he had learned deep learning from scratch. He expected it would be easy to catch up, so he jumped right in.

OpenAI has been publishing papers on the “Scaling laws” of generative AI. These “laws” describe the relationship between the amount of input data and performance. In this case, the more data, the better the performance.

“The more data we put into generative AI, the better it gets. That is guaranteed in the future. Of course, it is a lot of work because of the scale, but algorithmically it is only an extension of what has been done so far. There is no special creativity; everyone is doing pretty much the same thing," says Akiba, who feels that this direction will not fulfill his ambition to create a unique program. That is why he shares Sakana AI's vision of building models that break away from the Scaling Law, committing to a completely different approach. “We can compete based on people’s ingenuity rather than the amount of data we put in. It is putting my own skills to the test," Akiba says with “fun” written on his face.

Finding an interest that you can immerse yourself in

“I did not expect my university studies to be useful in the future. Yet, they have been useful even after I entered the workforce and changed my field of specialization. Reading papers critically in the right way and having to gather, systemize, and think about papers have been very useful in the end. University was a great opportunity to hone my basic skills in a topic I was passionate about. I hope students find an interest that they can immerse themselves in.”

※Year of interview 2024
Text/HORIBE Naoto
Photo/KAIZUKA Junichi

AKIBA Takuya

Research scientist, Sakana AI

He received his PhD from the Graduate School of Information Science and Technology at the University of Tokyo in 2015. He joined Sakana AI following research positions at NII (National Institute of Informatics), Preferred Networks, and Stability AI. Currently, he is managing various research projects on generative AI models. He has also co-authored a book about deep learning competitions organized by Kaggle (published by Kodansha).