Monthly:March 2019

Identifying and eliminating bugs in learned predictive models

One in a series of posts explaining the theories underpinning our research. Bugs and software have gone hand in hand since the beginning of computer programming. Over time, software developers have established a set of best practices for testing and debugging before deployment, but these practices are not suited for modern deep learning systems. Today, the prevailing practice in machine learning is to train a system on a training data set, and then test it on another set. While this reveals the average-case performance of models, it is also crucial to ensure robustness, or acceptably high performance even in the worst case. In this article, we describe three approaches for rigorously identifying and eliminating bugs in learned predictive models: adversarial testing, robust learning, and formal verification.Machine learning systems are not robust by default. Even systems that outperform humans in a particular domain can fail at solving simple problems if subtle differences are introduced. For example, consider the problem of image perturbations: a neural network that can classify images better than a human can be easily fooled into believing that sloth is a race car if a small amount of carefully calculated noise is added to the input image.Read More

Four tools to help drive your nonprofit’s mission forward

Google for Nonprofits started as a conversation about a volunteer program in an elevator in our New York office in 2010. Today, Google for Nonprofits has connected more than 200,000 nonprofits around the globe with Google products to help them run their organizations.

We believe that technology can do good in the world, which is why we offer eligible organizations access to a suite of Google products at no cost. Here’s a look at what each of the Google for Nonprofits products has to offer:

Bring teams together with G Suite for Nonprofits

G Suite is a set of integrated apps, including Gmail, Docs, Calendar, Drive and Hangouts Meet. It helps organizations with productivity, collaboration, and security—with G Suite, teammates can work from anywhere, on any device and focus their time on making an impact. If your nonprofit works across different locations, you can use G Suite to organize online video conferences on Hangouts, create group chats, work together simultaneously on the same document and much more.

Team Rubicon UK is a disaster response organization that needs to be prepared for and quickly respond to crises around the world. Using Google Forms and Sheets, Team Rubicon can go from 1,000 potential volunteers to a group of 12 skilled volunteers on the ground supporting a crisis within 24 hours.

Reach more supporters with Google Ad Grants

Google Ad Grants connects people to causes with $10,000 per month in free Google Advertising. Nonprofits can create ads to raise awareness, attract donors and help people around the world.

American Humane, the first national humane group in the U.S., uses Ad Grants to increase the organization’s exposure and educate people on their work in the animal welfare space. In one year, Google Ads drove over 1,000 new people to donate or sign up to volunteer and over 600 people to complete a service dog application.

Show your nonprofit’s impact with Google Maps & Earth

Google’s mapping tools help nonprofits in a wide range of ways, from tracking and sharing an organization’s impact to taking supporters on a virtual tour. To help organizations get started, Google for Nonprofits provides Google Maps Platform credits to nonprofits.  

iNaturalist, part of the California Academy of Sciences in San Francisco, uses Google Maps Platform to encourage environmental stewardship. It uses crowdsourcing, maps and photo sharing to gather data that helps conversationalist save a species before it becomes extinct. iNaturalist has succeeded in its goal of getting hundreds of thousands of people to connect with the natural world and continues to help scientists discover crucial findings about species.

Engage your supporters using video with the YouTube Nonprofits Program

YouTube amplifies your nonprofit’s reach by telling your story to a global audience.  Through the YouTube Nonprofit program, you’ll get access to Link Anywhere Cards that direct your viewers to your external campaign landing pages and to Creator Academy lessons tailored to nonprofits. Your nonprofit can also get access to the YouTube Space in Los Angeles or New York to shoot or edit your videos at YouTube’s creator studio. Just visit Space LA or Space NY and click the “Apply Now” button.

KMVT 15 is an award-winning nonprofit television station and media center that provides a community forum, media literacy education, hands-on training and civic engagement to local residents. KMVT 15 used YouTube for Nonprofits to engage its audience at a global scale: their YouTube channel has garnered more than 4.5 million views and seven thousand subscribers with a minimal marketing budget.

We’re inspired every day by the nonprofits across more than 50 countries who use our products to advocate their causes and make a real difference. If you’re interested in how Google’s technology can help your nonprofit, click here tolearn more.

Read More

Take action and stay up-to-date with dynamic email in Gmail

Over the past decade, our web experiences have changed enormously—evolving from static flat content to interactive apps. Yet email has largely stayed the same with static messages that eventually go out of date, or are merely a springboard to accomplish a more complex task. If you want to take action, you usually have to click on a link, open a new tab and visit another website.

Starting today, we’re making emails more useful and interactive in Gmail. Your emails can stay up to date so you’re always seeing the freshest information, like the latest comment threads and recommended jobs. With dynamic email, you can easily take action directly from within the message itself, like RSVP to an event, fill out a questionnaire, browse a catalog or respond to a comment.

Take commenting in Google Docs, for example. Instead of receiving individual email notifications when someone mentions you in a comment, now, you’ll see an up-to-date thread in Gmail where you can easily reply or resolve the comment, right from within the message.


Businesses have also already started using dynamic email to make their emails more actionable and relevant, like, Despegar, Doodle, Ecwid, Freshworks, Nexxt, OYO Rooms, Pinterest and redBus. Check out examples below—you’ll start to see these dynamic emails in the next few weeks.

First, Pinterest has made it easier to discover new ideas and save them to boards:


Next, with OYO Rooms, you can browse recommended hotels and rentals, and view details in fewer clicks right from the email:


And with Doodle’s dynamic email, you can respond to meeting without opening another website:


Like the rest of Gmail, dynamic email is safeguarded by best-in-class privacy and security protections. To ensure added security, those who want to send dynamic email have to be reviewed by Gmail first before they can get started.  

Dynamic emails will begin rolling out to Gmail users on the web today. Mobile support is coming soon. If you’re a G Suite customer, you will be able to enable dynamic email for your organization in the Admin console in the next few days. Heads up: if you’re using another mail app with Gmail, you’ll simply see the static version, and you can always revert back to it in Gmail, if you prefer.

Lastly, if you’re a developer and would like to learn how to build, test and send AMP emails, check out our documentation or this blog post for inspiration.

Read More

Build your next iOS and Android app with Flutter

Mobile development is full of compromises. When you’re building a new app, you’re often forced into a difficult choice: do you build the same app twice—once for iOS, once for Android—so that you have a high quality experience for both platforms? Or do you create one app from a shared codebase that works across both platforms, but doesn’t have the performance or user experience you were hoping for? Flutter offers a third way: enabling high-quality user experiences with excellent performance, along with letting you express your designers’ intent and share a common codebase.

Whether you’re an entrepreneur with a new app idea, a developer who’s frustrated by the edit-compile-debug cycle of building an app, or a designer who wants to be able to iterate on new design concepts, here are a few reasons why you should consider creating your next app with Flutter.

Beautifully-designed apps on each platform

Creating app designs can involve trade-offs between the creative intent of a designer and the reality of what actually goes into production. Mobile designers and developers often work in different worlds, separated by a hard boundary between the different tools that they use and challenges with iterating on a design during the development process. Sometimes the designer’s vision is compromised by limitations in the APIs or framework the developer uses; sometimes, visual polish gets deferred until “later” because of other development work (and “later” often means “never” in practice).

Two mobile phone screens showing the differences between what was designed, and what was sent into production.

With Flutter, you have control over every pixel on the screen from the beginning, including a full set of widgets that deliver pixel-perfect experiences on both iOS and Android. Designers are using Flutter to create attractive experiences like Reflectly, the number one journaling app on the Apple app store:

The user interface of Reflectly, a journaling app.

There are also some design tools made for Flutter—like 2Dimensions Flare, which you can use to build animations and incorporate them into any app with a single line of code. Here’s an example of a custom animation built with Flare that interactively follows your typing at login:

A mobile log-in screen displaying a bear who follows your cursor as it moves across the screen.

You can find more details about this example here.

And Supernova, a design-to-code tool, recently announced support for exporting Sketch designs directly to Flutter, allowing users of this popular design and wire-framing tool to turn their ideas directly into code.

Fast apps on each platform

Rather than introducing a layer of abstraction between your code and the underlying operating system, Flutter apps are native apps—meaning they compile directly to both iOS and Android devices.

Flutter’s programming language, Dart, is designed around the needs of apps that are created for global audiences. It’s easy to learn, contains a comprehensive set of libraries and packages that reduce the amount of code you have to write and is built for developer performance. When you’re ready to release your app, you can compile your code directly to the ARM machine code of your phone—meaning what you write is exactly what appears on the device—so you can harness the full power of your phone, rather than using a language like JavaScript that needs a separate engine to run.

A side-by-side comparison of two mobile screens showing Flutter rendering animations on the phone in real-time.

Flutter rendering animations on the phone in real-time.

Flutter isn’t a games engine, but it brings games-level performance to your application. Every pixel in Flutter is drawn with the Skia graphics engine: the same hardware-accelerated engine that powers Android and Chrome. This combination enables fast, glitch-free performance for apps—meaning that apps can operate on a phone’s screen at 60 frames per second—which will never feel sluggish, even on a slower device.

Productive app creation on each platform

If you’re a mobile app developer, you might feel like you spend more time waiting than coding. When you make a change to your code, you have to recompile it, deploy it to a mobile phone and then bring the app back into the same state it was in before you made the change to see the results. By contrast, Flutter introduces a new capability called Stateful Hot Reload, which transforms this development cycle, letting you implement changes in milliseconds—not minutes. Stateful Hot Reload also allows better collaboration between developers and designers when they want to improve the app design and immediately see the effects.

Two side-by-side screens show how an app’s user interface and logic are updated in the app without the need for recompilation.

Changes to an app’s user interface and logic are updated in the app without the need for recompilation.

Teams using Stateful Hot Reload report major gains to their productivity when making apps. Combining the quick turnaround on changes with the ability to ship for both iOS and Android, we’ve seen apps from brands like Abbey Road Studios, Hamilton and Reflectly go from writing their first line of code to a published app in just weeks.

Get started today

We just launched version 1.2 of Flutter at Mobile World Congress 2019. We encourage you to try Flutter by visiting You’ll find all the resources you need to get started including videos, codelabs, case studies, documentation and community links.

Read More

Supporting choice and competition in Europe

For nearly a decade, we’ve been in discussions with the European Commission about the way some of our products work. Throughout this process, we’ve always agreed on one thing一that healthy, thriving markets are in everyone’s interest.

A key characteristic of open and competitive markets一and of Google’s products一is constant change. Every year, we make thousands of changes to our products, spurred by feedback from our partners and our users. Over the last few years, we’ve also made changes一to Google Shopping; to our mobile apps licenses; and to AdSense for Search一in direct response to formal concerns raised by the European Commission.  

Since then, we’ve been listening carefully to the feedback we’re getting, both from the European Commission, and from others. As a result, over the next few months, we’ll be making further updates to our products in Europe.

Since 2017, when we adapted Google Shopping to comply with the Commission’s order, we’ve made a number of changes to respond to feedback. Recently, we’ve started testing a new format that gives direct links to comparison shopping sites, alongside specific product offers from merchants.  

On Android phones, you’ve always been able to install any search engine or browser you want, irrespective of what came pre-installed on the phone when you bought it. In fact, a typical Android phone user will usually install around 50 additional apps on their phone.

After the Commission’s July 2018 decision, we changed the licensing model for the Google apps we build for use on Android phones, creating new, separate licenses for Google Play, the Google Chrome browser, and for Google Search. In doing so, we maintained the freedom for phone makers to install any alternative app alongside a Google app.

Now we’ll also do more to ensure that Android phone owners know about the wide choice of browsers and search engines available to download to their phones. This will involve asking users of existing and new Android devices in Europe which browser and search apps they would like to use.

We’ve always tried to give people the best and fastest answers一whether direct from Google, or from the wide range of specialist websites and app providers out there today.  These latest changes demonstrate our continued commitment to operating in an open and principled way.

Read More

Stadia: a new way to play

For 20 years, Google has worked to put the world’s information at your fingertips. Instant delivery of that information is made possible through our data center and network capabilities, and now we’re using that technology to change how you access and enjoy video games.

Stadia is a new video game platform, delivering instant access to your favorite games on any type of screen—whether it’s  a TV, laptop, desktop, tablet or mobile phone. Our goal is to make those games available in resolutions up to 4K and 60 frames per second with HDR and surround sound. We’ll be launching later this year in select countries including the U.S., Canada, U.K. and much of Europe.

To build Stadia, we’ve thought deeply about what it means to be a gamer and worked to converge two distinct worlds: people who play video games and people who love watching them. Stadia will lift restrictions on the games we create and play—and the communities who enjoy them.

Advanced game streaming

Using our globally connected network of Google data centers, Stadia will free players from the limitations of traditional consoles and PCs.

When players use Stadia, they’ll be able to access their games at all times, and on virtually any screen. And developers will have access to nearly unlimited resources to create the games they’ve always dreamed of. It’s a powerful hardware stack combining server class GPU, CPU, memory and storage, and with the power of Google’s data center infrastructure, Stadia can evolve as quickly as the imagination of game creators. 

Data centers make Stadia possible, but what sets the system apart is how it works with other Google services. In a world where there are more than 200 million people watching game-related content daily on YouTube, Stadia makes many of those games playable with the press of a button. If you watch one of your favorite creators playing Assassin’s Creed Odyssey, simply click the “play now” button.Seconds later, you’ll be running around ancient Greece in your own game/on your own adventure—no downloads, no updates, no patches and no installs.

But what’s a gaming platform without its own dedicated controller? Enter the Stadia controller*.

The Stadia controller

When we designed the Stadia controller, we listened to gamers about what they wanted in a controller. First, we made sure to develop a direct connection from Stadia controller to our data center through Wi-Fi for the best possible gaming performance. The controller also includes a button for instant capture, saving and sharing gameplay in up to 4k of stunning resolution. And it comes equipped with a Google Assistant button and built-in microphone.

Using Google’s vast experience, reach and decades of investment we’re making Stadia a powerful gaming platform for players, developers and YouTube content creators—of all sizes. We’re building a playground for every imagination.

*This device has not been authorized as required by the rules of the Federal Communications Commission. This device is not, and may not be, offered for sale or lease, or sold or leased, until authorization is obtained.

Read More

Spin your agency into the future

Anass Koudiss | Global Education Programme Manager, Leadership

Reading time: 3 minutes

If you’ve read a newspaper or an advertising industry magazine in the last few weeks, it’s very likely that machine learning and automation were mentioned at least once. They’re the hottest new topics in the marketing world. 

As the tech evolves, and manual tasks move towards automation, it’s up to agency leaders to make sure it’s being used to provide new value to their clients and spur agency growth. 

Since every agency is unique, each will need to take its own approach. Every agency has its own strengths, growth opportunities and resource capabilities. What many agencies have in common, though, is their powerful ability to grow and transform quickly in an industry that’s constantly changing. 

At Google, we’re powering our advertising solutions with machine learning. Automated solutions like Smart Bidding, Smart Creatives, and App campaigns take care of routine tasks, freeing up your team to focus on ad-testing, advanced insights, or more face time with clients. 

As with any big change, adopting this new approach takes a lot of planning. So we developed a powerful framework called “The Web of Transformation”, to help you envision the future of your unique agency. To get started, bring your leadership team together, print out this A3 poster, and spin your agency web with the following steps: 

1. To begin, choose the eight most important dimensions that make up your agency. Specifically, pick the eight things that help you and your clients grow. 

2. Feel free to choose from the suggestions here, but make sure you pick dimensions that make sense for your agency. 

3. Next, with the help of your team, rank each of the eight dimensions from most important to least. 

4. Finally, grade your ability in each dimension on a scale of 1 to 5. Here’s an example of what it should look like. 

5. Make sure to evaluate and analyse where you excel, and where you need to step up your game. Ask yourselves the questions below to help drive the discussion. 

Together, we can use this process to capitalize on automation and Machine Learning opportunities, as we’ve done before with mobile, multi-screen, and video. So weave your web carefully, it will give you a glimpse into the future of your agency.

Read More

Kormo, an app helping young job seekers find work, launches in Jakarta

I recently traveled to Jakarta, Indonesia, where I met with job seekers just starting their careers. Repeatedly, they shared that finding entry-level and part-time jobs is confusing and difficult, especially when you don’t have much in the way of work experience, professional contacts or resources. To help young Indonesians connect with employers, learn new skills and build their careers, we’re bringing the Kormo app to Jakarta. 

kormo team in jakarta

Speaking to young job seekers in Jakarta who had just spent the day at the mall going from shop to shop to drop off their CVs.

Kormo is one of many projects incubated inside of Area 120, Google’s workshop for experimental projects. Initially launched in Dhaka—the capital city of Bangladesh and one of the most densely populated cities in the world—Kormo has matched people with more than 25,000 jobs, including retail sales, delivery and contact center positions. Our time in Dhaka and Jakarta taught us a lot about the needs of young job seekers and employers in markets where rapid urbanization and growth are impacting labor markets.

Making career-building easy and assistive


A group of university students in Dhaka, creating their digital CVs on Kormo

Job seekers told us that they weren’t sure what jobs were available to them, as listings typically travel by word-of-mouth or aren’t posted online. And though many of these seekers pay someone to create a paper CV (or résumé), their busy schedules and traffic congestion in the city makes it inconvenient to drop hard copies in employer CV boxes. Even after they apply, many seekers never hear back on the status of their candidacy.

Kormo lets anyone to build a digital CV quickly, and for free. The digital CV updates dynamically as job seekers find work or enroll in trainings through the Kormo app. Kormo will also display open job listings in the city—the job opportunities listed will reflect the job seeker’s growing profiles and skills. Meanwhile, we partner with employers, ranging from large corporates to small or medium enterprises, to include their jobs in Kormo’s marketplace and enable seekers to apply directly from our app.

Kormo job list

Providing access to skills training

As the job markets evolve in Bangladesh and Indonesia, employers are looking for candidates with new skill sets, including proficiency with new technologies. While training resources exist both online and offline, job seekers told us that they weren’t sure about whether a specific program could help them get a job. Plus, many of the programs that offer formal certificates cost money and require time commitments that can be hard to sustain.

kormo video training

Skills development expert and Kormo partner, Don Sumdany, creating localized videos to help people who use Kormo ace their next interview.

Within the Kormo app, we built a Learn tab where seekers can access relevant, up-to-date and free training content in the form of videos, articles and courses based on individual interests.


While learning from job seekers in Dhaka, we met Abida, a college student beginning her career in the city. Kormo recently matched her to her first job at a well-known retail store, and she now has the financial stability she needs to continue her education. We hope to see many more stories like Abida’s emerge now that Kormo is available in another city.


Abida, a college student  in Dhaka, at her new job that she found through Kormo.

With Kormo, our ultimate goal is to provide economic opportunity and mobility to more people in countries like Bangladesh or Indonesia. If you live in one of these countries, you can download the Kormo app for free from the Google Play store. For businesses looking to hire through Kormo, please sign up from our site.

Read More

At SXSW, creative agencies partner with Ad Council and YouTube to spark stories for a cause

What do Ozzy Osbourne and Prince Charles have in common? A lot, if you’re looking at demographics alone; they’re both male, born in the same year, and from England. Yet, as has been pointed out many times before, identifiers like age and gender don’t give the full picture. 

For SXSW, we’ve partnered with 6 world-class creative agencies to explore how audience signals – beyond demographic data – can enable a spark for better, more relevant storytelling. 

Partnering with Ad Council on three causes – empowering girls in STEM, preparing for disasters, and ending hunger – we challenged each agency to each develop two different creatives that would resonate with the intended audience they wanted to engage, from sports fans to technophiles

The 12 campaigns, showcasing how customized creative can ignite conversations and action around important causes, will debut at SXSW on March 9th. 

Check out the full playlist to or dive in more to the story behind the work below. 

To Empower Girls in STEM 

Research shows that young girls are interested in STEM subjects but get discouraged from pursuing them by outdated stereotypes(1). AMP Agency and Hecho Studios took on the challenge to create ads that would speak to key audiences for She Can STEM and encourage them to empower young women in their lives in STEM.

AMP Agency 

While it’s often difficult to know what leads a girl down the path to success, AMP Agency recognized that parents and other role models may play the most important role in guiding their futures. A core message unites the two creatives; you can set a young woman’s future in motion by clueing into her interests and nourishing her passion for STEM. 

Watch the ad designed for tech-savvy people

Watch the ad designed for bargain hunters.

Hecho Studios 

To speak to parents of grade schoolers as well as avid readers, Hecho Studios developed an eye-catching showcase of women who have pioneered STEM fields. 

Watch the ad designed for parents of grade schoolers
Watch the ad designed for avid readers

To Drive Disaster Preparedness 

While the thought of preparing for a natural disaster can be overwhelming, developing an emergency plan can save lives. We Are Unlimited and Energy BBDO partnered with Ad Council to inspire more people to develop household disaster plans. 

Energy BBDO 

Recognizing that pet-lovers tend to watch humourous videos skiwing toward dog content, Energy BBDO created a funny spot featuring a conversation between a dog and owner. For DIYers, Energy BBDO developed high-action creative after seeing how this audience over indexed on action packed videos. 

Watch the ad designed for pet lovers

Watch the ad designed for DIYers who enjoy building things on their own.

We Are Unlimited 

Building different executions from a core campaign idea, We Are Unlimited created two day-in-the-life scenarios to show how an average day for stock-market savvy people versus travel buffs could go awry when disaster strikes. 

Watch the ad designed for stock-savvy investors

Watch the ad designed for travel buffs.

To Combat Hunger 

Hunger affects one in eight Americans(2), yet many are unaware of the prevalence. For this cause, BBDO and Grey created ads for four key audiences to drive donations to Feeding America. 


Creating one central concept that depicts the reality of impoverished meals, BBDO tailored the tone and details of the creative executions to speak to each audience. 

Watch the ad designed for gamers.

Watch the ad created for outdoor enthusiasts.


Grey’s overarching creative concept spoke to how hunger can hide in plain sight. Each creative execution featured a unique call-to-action to inspire action from that audience. 

Watch the ad designed for sports fans.

Watch the ad designed for green living enthusiasts.


(1)  Source:
(2) Citation: Alisha Coleman-Jensen, Matthew P. Rabbitt, Christian A. Gregory, and Anita Singh. 2018. Household Food Security in the United States in 2017, ERR-256, U.S. Department of Agriculture, Economic Research Service

Read More

TF-Replicator: Distributed Machine Learning for Researchers

At DeepMind, the Research Platform Team builds infrastructure to empower and accelerate our AI research. Today, we are excited to share how we developed TF-Replicator, a software library that helps researchers deploy their TensorFlow models on GPUs and Cloud TPUs with minimal effort and no previous experience with distributed systems. TF-Replicators programming model has now been open sourced as part of TensorFlows tf.distribute.Strategy. This blog post gives an overview of the ideas and technical challenges underlying TF-Replicator. For a more comprehensive description, please read our arXiv paper.A recurring theme in recent AI breakthroughs – from AlphaFold to BigGAN to AlphaStar – is the need for effortless and reliable scalability. Increasing amounts of computational capacity allow researchers to train ever-larger neural networks with new capabilities. To address this, the Research Platform Team developed TF-Replicator, which allows researchers to target different hardware accelerators for Machine Learning, scale up workloads to many devices, and seamlessly switch between different types of accelerators.Read More