I don’t know about you, but when I play video games, I actually like to, you know, play them. If I wanted to outsource gameplay to someone else, I’d watch a let’s play or a Twitch stream or something. But Google is working on an AI model that has the ability to play video games on your behalf, so long as you tell it what you want it to do: It’s called SIMA, short for Scalable Instructable Multiworld Agent, and if it works as advertised, AI might just be taking over your favorite hobby.
Google DeepMind, the company’s artificial intelligence department, announced the new model in a blog post as well as a post on X (formerly Twitter). SIMA, according to Google DeepMind, is the first generalist AI agent that can follow natural language instructions in 3D environments. In other words, it can play video games based on your commands. You say “turn left,” and SIMS turns the character left.
Google DeepMind worked with eight video game studios in order to train SIMA, including No Man’s Sky’s Hello Games and Teardown’s Tuxedo Labs. The development team wanted as many different types of games as possible to train SIMA with, since each new variable added another skill to the model’s abilities. Google DeepMind even built a sandbox-like environment, where SIMA would need to build structures to test its understanding of physics and object manipulation.
What makes SIMA so successful, at least in theory, is that it doesn’t need any technical information about the video game itself such as source code or APIs. It can act based on just the video games’ images and your natural language commands. Google DeepMind says SIMA can execute over 600 “basic skills,” such as turning in a specific direction, interacting with objects, and using the game’s menus. That said, Google DeepMind is still working on more complication actions, as well as commands that contain multiple sub-tasks. It’s one thing to tell the AI to climb the ladder in front of them, but it’s a whole other beast to train it to respond accurately to “mine for resources to build shelter.” The company says this is a limitation with large language models in general—bots will respond to simple commands, but struggle with performing intuitive actions independently.
In the meantime, Google DeepMind is touting its success with its multi-game training model, claiming that SIMA outperforms models trained on one specific game at a time. In fact, the company says that SIMA can respond better in a game it’s never seen before than a model that was trained only on that game.
While SIMA isn’t available to the public yet, you can imagine some potential use cases for the tech. I imagine this could be a great accessibility option in the future: For players who have trouble using traditional controllers, telling the bot how to control the player could be a game-changer. Of course, Google’s end goal here seems to be beyond this situation, in that they want the AI to be able to play the games by itself. That could be great for bypassing repetitive tasks, like leveling up or earning money, but it also begs the question: Why are you playing the game in the first place if you want a robot to do all the playing?
This is Google’s second big foray into AI gaming: Last month, we learned the company was working on a model that could generate 2D platformers from natural language commands as well. Perhaps sometime in the near future, the company will roll out Google Gaming: Just tell the AI what type of game you’d like to see, and it’ll both generate and play the game for you in real time. How fun.
