Research Scientist, Gemini, UI Control

Snapshot

Artificial Intelligence could be one of humanity’s most useful inventions. At Google DeepMind, we’re a team of scientists, engineers, machine learning experts and more, working together to advance the state of the art in artificial intelligence. We use our technologies for widespread public benefit and scientific discovery, and collaborate with others on critical challenges, ensuring safety and ethics are the highest priority.

About Us

There is great interest in having AI agents execute complex and long-running tasks. For example, instead of merely creating a travel itinerary, an agent could read reviews, compare options, and finds flights, accommodations, and activities. To be able to perform all actions a human can do, it is likely our agents will need to be able perform actions via a user interface. The Gemini UI Control group is dedicated to enabling computers to understand and interact with user interfaces in a human-like manner, thereby unlocking a wide range of agentic capabilities.

We develop models that power UI Control applications across Google, including Project Mariner and the AI Studio Computer Use API, with more on the horizon.. Our primary focus is on post-training generative models to equip them with the ability to comprehend and interact with user interfaces to achieve specific goals

The Role

We concentrate on the following three key areas related to UI understanding and control:

• Data Enhancement: Adding new data to post-training mixtures in Gemini


• Self-Improvement: Exploring methods such as online exploration and reinforcement learning


• Evaluation Contributions: Developing and contributing new evaluation methodologies

The objective is to continuously improve quality while tackling tasks of increasing complexity and ambiguity across a wider range of surfaces.

Key responsibilities:

This role involves making core contributions to post-training in Gemini that advances the frontier of what digital agents can do and is deployable into production. The employee will identify model areas of improvement and own work related to training and evaluating UI Control agents across various digital environments. Multi-step reinforcement learning in multimodal, digital environments is a focus.  Research that improves the ability of the agent to handle more complex tasks or incorporate more user interaction is a plus.

About You

The ideal candidate is someone who can push the boundaries of AI research related to computer use models and is comfortable working across the research stack to see their work through.

In order to set you up for success as a Research Engineer at Google DeepMind,  we look for the following skills and experience:

• PhD or relevant experience in computer science related field.


• Experience with multi-step reinforcement learning


• Experience with post-training generative models

In addition, the following would be an advantage: 

• Understanding of model training, evaluation metrics


• Experience with designing and analyzing machine learning experiments


• Ability to design and build evaluations


• Adaptability: The field is evolving rapidly, requiring a willingness to learn new technologies and approaches.

The US base salary range for this full-time position is between $166,000 - $244,000 + bonus + equity + benefits. Your recruiter can share more about the specific salary range for your targeted location during the hiring process.

At Google DeepMind, we value diversity of experience, knowledge, backgrounds and perspectives and harness these qualities to create extraordinary impact. We are committed to equal employment opportunity regardless of sex, race, religion or belief, ethnic or national origin, disability, age, citizenship, marital, domestic or civil partnership status, sexual orientation, gender identity, pregnancy, or related condition (including breastfeeding) or any other basis as protected by applicable law. If you have a disability or additional need that requires accommodation, please do not hesitate to let us know.