Google News App

---

John,

Thanks to you, we have come an amazing distance together since you first joined the team now five and a half years ago. Huge thanks to you for your partnership along the way as we’ve advanced Waymo’s state-of-the-art autonomous driving tech, our autonomous ride-hailing and delivery businesses, and your guiding hand as we chart our ambitious future. We’re grateful you’ll be staying on as an advisor.

Waymonauts,

As we start this next chapter together, we could not be more excited and grateful to have the opportunity to lead this team and company as your co-CEOs. We’re committed to working alongside you to build, deploy, and commercialize the Waymo Driver and drive the success of our incredible team and this company. We’re energized by the road and opportunity ahead of us.

Thank you for your passion, your world-class contributions, and your confidence in us as we lead Waymo forward together.

– Dmitri & Tekedra

One of the most important things an intelligent driver needs to do is to understand what the road users around it are going to do next. Is that pedestrian trying to cross the street? Is that car parallel parked, or about to pull into my lane? Will that speeding vehicle stop at the stop sign?

Accurately predicting the behavior of other road users is one of the hardest problems in autonomous driving. It also has significant safety implications – correctly evaluating another driver’s likely behavior is essential to being able to mitigate crashes.

While AV researchers have made significant progress on the motion forecasting problem in recent years, more high-quality open-source motion data can help achieve new breakthroughs throughout the industry.

Today, we’re expanding the Waymo Open Dataset with the publication of a motion dataset – which we believe to be the largest interactive dataset yet released for research into behavior prediction and motion forecasting for autonomous driving.

We’re also announcing the next round of Waymo Open Dataset Challenges – with cash awards – to help encourage new research into both perception and behavior prediction.

Finally, we’re releasing a paper describing the state-of-the-art research offboard perception method we used to annotate the motion dataset, so any research team can consider our techniques when exploring ways to create their own high-quality motion data.

The most useful data for hard motion forecasting challenges

High-quality motion data is particularly valuable because it is time-consuming and expensive to source.

Generating a motion dataset with high-quality labels requires a sophisticated perception system that can accurately pick out agents and objects from camera and lidar data, and track their movements throughout a scene.

Interesting motion data is also hard to gather. Most day-to-day driving is uneventful – which makes for uninformative data when you are building a system to predict what could happen on the road in unusual situations. As a result, existing datasets often have a limited number of interesting interactions.

The Waymo Open Dataset seeks to address these challenges. With object trajectories and corresponding 3D maps for over 100,000 segments, each 20 seconds long and mined for interesting interactions, our new motion dataset contains more than 570 hours of unique data. We believe it’s the largest dataset of interactive behaviors yet released for autonomous driving research.

We have tried to make this data as useful as possible for researchers exploring how to create effective behavior prediction systems:

1. We have included a wide variety of data to help develop more robust, flexible motion forecasting models. The Waymo Open Dataset is one of the most geographically varied motion dataset yet released, featuring a wide variety of road types and driving conditions captured around the clock in different urban environments, including San Francisco, Phoenix, Mountain View, Los Angeles, Detroit and Seattle, to encourage models that can better generalize to new driving environments.
2. We have mined the data specifically to include interesting examples of agents interacting – whether it’s cyclists and vehicles sharing the roadway, cars quickly passing through a busy junction, or groups of pedestrians clustering on the sidewalk. At 20 seconds, each segment is long enough to train models that can capture complex behaviors. We have also included map information for each scene to provide semantic context that’s important for accurate predictions.
3. We used our state-of-the-art research offboard perception system to create higher-quality perception boxes than are found in other datasets. This is important because the better the perception system is at tracking the objects and agents in the motion data, the more accurate the resulting behavior prediction model will be at predicting what they are going to do.
4. We are releasing a paper describing the state-of-the-art offboard perception techniques we used to annotate this new data. We trained this system on the perception data that we already made available in the Waymo Open Dataset, so our process for creating this new motion dataset is as transparent as possible. Not only should this make it easier for researchers to scrutinize the quality of the data we’re releasing, but it may also inspire others as they consider ways to create their own high-quality motion datasets, too. This paper has been accepted at CVPR 2021.
5. We have created new benchmarks for assessing behavior prediction models. High-quality evaluation is critical to making progress in machine learning: the better the benchmark, the better the model that scores well on it. So we have released a suite of interaction metrics to allow robust benchmarking for models trained on this or any other motion dataset.

We hope this data will be useful for researchers working on behavior prediction in a wide variety of domains and industries. You can read more about the dataset on our website and in our GitHub repo, and download the data here. We have also released a Colab tutorial to provide an accessible introduction to using the dataset. We will release an accompanying paper explaining the main features of the dataset soon.

 

Announcing four new Waymo Open Dataset challenges

Alongside our latest dataset, we are delighted to announce the next round of Challenges to encourage work on both perception and behavior prediction. The four challenges are:

1. Motion prediction challenge: Given agents’ tracks for the past 1 second on a corresponding map, predict the positions of up to 8 agents for 8 seconds into the future.
2. Interaction prediction challenge: Given agents’ tracks for the past 1 second on a corresponding map, predict the joint future positions of 2 interacting agents for 8 seconds into the future.
3. Real-time 3D detection: Given lidar range images and their associated camera images, produce a set of 3D upright boxes for the objects in the scene, with a latency requirement.
4. Real-time 2D detection: Given a set of camera images, produce a set of 2D boxes for the objects in the scene, with a latency requirement.

The winning team for each challenge will receive a $15,000 cash award, with second-place teams receiving $5,000 and third place $2,000.

You can find the rules for participating in the challenges here. The challenges close at 11:59pm Pacific on May 31, 2021, but the leaderboards will remain open for future submissions. We’re also inviting eligible winners to present their work at our Workshop on Autonomous Driving at CVPR, scheduled for June 20, 2021.

Since we first launched the Waymo Open Dataset in August 2019, it has had a significant impact on the research community:

• Its high-quality perception data has already spurred much academic research—as signaled by over 130 citations to our perception paper—benefitting autonomous technology teams worldwide.
• More than 150 teams participated in last year’s Challenges.
• It is also encouraging work far beyond the Challenges. Just recently, a collaboration with researchers at Google Brain led to the publication of a set of 3D scene flow annotations, taking advantage of the vast quantity and quality of annotated LiDAR data in the Waymo Open Dataset to create a set of 3D scene flow labels 1,000x larger than previous real-world datasets, to encourage further research on 3D motion estimation.
• It has also lowered barriers to entry for students and academics to experiment with new techniques without needing an autonomous vehicle to gather their own data.

We hope this expansion into motion data spurs on a new wave of research.

I know first-hand how incredibly important high-quality data is to tackling thorny AV research questions, and I’m incredibly proud of the Waymo research team’s efforts to continue evolving the dataset and making it applicable for an increasing number of research areas. We welcome community feedback on how to make our dataset even more impactful in future updates.

I’ve spent over 20 years working in crash avoidance research, in the belief that improved driving technology is the key to reducing these needless deaths.

But while we often hear that autonomous driving technology could make a dramatic difference, before today there has not been a published scenario-based study that we’re aware of that looks into how autonomous technology performs in scenarios that led to fatal crashes by human drivers.

Today, we’re releasing the results of a study into how the Waymo Driver might perform in such tragic situations, which builds on the research that we released in October. While the October study showed that the Waymo Driver was only involved in minor collisions over more than 6 million miles driven in reality on public roads, our most recent study shows how the Waymo Driver likely would have performed in the majority of fatal crashes that occured on the same roads over a 10 year period. The results are encouraging.

NHTSA statistics show that Maricopa County, which encompasses our service area, consistently makes the list of highest annual pedestrian fatality rate each year, demonstrating the unique focus on pedestrian crashes while driving on those streets. For our analysis, we collected information on every fatal crash that took place in Chandler, Arizona between 2008-2017*. We excluded crashes that didn’t match situations that the Waymo Driver would face in the real world today, such as when crashes occurred outside of our current operating domain. Then, the data was used to carefully reconstruct each crash using best-practice methods. Once we had the reconstructions, we simulated how the Waymo Driver might have performed in each scenario.

In total, the simulated Waymo Driver completely avoided or mitigated 100% of crashes aside from the crashes in which it was struck from behind, including every instance that involved a pedestrian or cyclist (20 simulations in total). This is the first time an autonomous technology company has shared its evaluation for how the system might perform in real-world fatal crash scenarios.

Figure 1: Results of the Waymo Driver’s performance in simulated tests

We started by simulating 72 fatal crashes as they occurred on public roads in our operating domain, which covers thousands of miles of road in southeast Phoenix. Since many of these crashes involved two vehicles, we ran separate experiments simulating the Waymo Driver in the role of each vehicle—first replacing the vehicle that initiated the crash (the “initiator”) with the Waymo Driver, and then replacing the vehicle that responded to the other vehicle’s actions (the “responder”) with the Waymo Driver. That left us with 91 simulations in total.

When we swapped in the Waymo Driver as the simulated initiator (52 simulations), it avoided every crash by consistent, competent driving, and obeying the rules of the road—yielding appropriately to traffic, executing proper gap selection, and observing traffic signals. Here, for example, you can see on the bottom of the screen that the simulated Waymo Driver avoids a reconstructed version of a real-life fatal crash by obeying the speed limit—and not running a red light, as the initiator did in real life:

Figure 2: Waymo Driver replacing the initiator. The Waymo Driver is driving the speed limit and stops at a traffic light.

But as we know, human drivers can make mistakes. They sometimes speed excessively through red lights, fail to yield on turns, and drive while tired, distracted or impaired. Since humans will be on the road for the foreseeable future, it’s important to understand how well autonomous driving systems perform in response to mistakes made by humans.

That’s one of the reasons why we ran simulations with the Waymo Driver in the role of the responder. If the Waymo Driver performed consistently better than the human drivers in these crashes, then that helps indicate the broader safety benefits our autonomous technology can achieve. In fact, that is exactly what our simulations suggest:

• When the Waymo Driver was placed in the responder role, it completely avoided 82% of simulated crashes. In fact, in the vast majority of events, it did so with smooth, consistent driving—without the need to brake hard or make an urgent evasive response.
• In another 10% of scenarios when the simulated Waymo Driver was the responder—all at an intersection when another vehicle turned across its path— it took action that mitigated the severity of the crash.
• Only 8% of responder crash simulations were unchanged. In all of these, the human-driven vehicle struck the rear of the simulated Waymo Driver when it was either stopped or traveling at a constant speed, giving the Waymo Driver little opportunity to respond.

In other words, even when a human driver did something to initiate a crash, such as running a red light, the simulated Waymo Driver avoided or mitigated the vast majority of these fatal crashes.

Replaying the same scenario discussed above, for example, the simulated Waymo Driver is approaching from the right of the screen, and has the right of way at a green light. But as it approaches the intersection, it spots the speeding car approaching from the bottom, predicts that it isn’t going to stop at the red light, and slows considerably until the speeder passes, avoiding the crash:

Figure 3: Waymo Driver replacing the responder. The Waymo Driver perceives and accurately predicts the initiator.

You can read more about our methodologies and our full results in our academic paper.

While these are early results, and we are cognizant that they only represent one of many indicators that should be used for assessing our performance under real-world circumstances, we believe they demonstrate the potential of AV technology to improve safety.

Given that the version of the Waymo Driver we tested in simulation is the same version that we currently have on the road, these results are a testament not only to the capabilities of the Waymo Driver, but also to the rigor of our more than a decade of experience and our safety readiness framework. Since 94% of crashes involve human error, we believe we have an opportunity to improve road safety by replacing the human driver with the Waymo Driver. This study helps validate that belief.

By sharing this data, we’re living up to our obligation to demonstrate the trustworthiness of the technology we’re building. We encourage other companies developing autonomous driving technology to do the same. We plan to continue providing additional evidence to allow people to reach their own conclusions about our safety readiness.

This research was made possible because of the thorough crash reports the Chandler Police Department and Arizona Department of Transportation (ADOT) prepare following fatal crashes, and we’re grateful to ADOT, Chandler Police Department, and Arizona Department of Public Safety for making these reports available to the public upon request. This ultimately helps make possible the improvements to roadway and vehicle design that have improved road safety over recent decades, and collaboration with public safety bodies is important for continued improvements.

The safety of autonomous driving technology is the deliberate result of careful development and thorough and continuous evaluation and refinement. This study provides yet more useful demonstration of its potential safety benefits.

I’m Jesse, and live in Tempe, Arizona. I love traveling and trying new things out; I’m originally from Charlotte, North Carolina and recently moved to Phoenix with American Airlines.

How did you hear about Waymo One?

I heard about Waymo from my roommate. He told me it was something I might like so I downloaded the app and called a car right away to see it for myself.

What was your first fully autonomous ride like? When did it start to feel like a normal part of your daily life?

My first fully autonomous ride was the very first ride I booked with Waymo. I thought it was very fun, and I smiled the whole time calling all my friends on FaceTime. I didn’t want to get out – I was having so much fun!

What do you primarily use Waymo One for?

I primarily use Waymo to get around and go to Walmart mostly. I just moved here, and most ride-hailing companies are expensive, but Waymo always has the best prices. The cars are also always clean and ready for me when they pull up.

What’s the most interesting thing that’s happened to you while riding with the Waymo Driver?

In the Waymo ride I just took, there was a bus that pulled over in front of us, and the car read that and knew when it was safe to go around. It did better than some people that have driver’s licenses. I was very impressed about it, and it just made me wonder if this car with no human driver can do it, why can’t most of the people on the road do it?

What would you say to people who might be hesitant to get in a car with no human driver?

Most people would be hesitant to get in a car when there is no one in the front seat, I’m not! And most people that commented on my Facebook video stated that they would be scared or would never do that, but I had a great experience and would hop in a Waymo ride before using any other service.

Autonomous vehicles promise to change this, revolutionizing our cities by making transport safer, cheaper, and easier to access. But deploying autonomous driving technology to get there – perhaps one of the greatest engineering problems of our time – isn’t possible without developing a deep understanding of the nature of the challenges it must master.

Building the most advanced technology stack for urban driving

In San Francisco, that means tackling the city’s iconic topographical variety – from rolling hills to sandy ocean highways, tiny side streets to huge freeways, bike lanes to tram tracks, and everything in between.

It also means learning how to handle the city’s other road users safely. San Francisco’s streets are busy with traffic, pedestrians, cyclists, scooters, and emergency vehicles – and that’s not to mention the cable cars, trolleys, streetcars and light rail vehicles that define the cityscape.

Building an autonomous driving service that can tackle this complexity safely and effectively is an enormous engineering challenge. Over the past decade, we’ve tested our technology in dozens of cities to build up the most mature technology stack in the industry – one advanced enough to power the US’s only public commercial ride-hailing service in Phoenix, Arizona, with no human controlling the vehicles either in-car or remotely.

Throughout that time, we’ve also been gearing our technology to drive in dense cities such as San Francisco. To give just a few examples:

• We’ve optimized the Waymo Driver’s 360° vision system and lidar to navigate the complexities of urban driving. Our highly sensitive cameras can spot traffic lights changing at a long distance – even among the papel picado on 24th Street – to enable smooth driving. And our cameras and lidar can instantly spot a jaywalker sprinting across our path and act appropriately – even when they emerge suddenly from behind a vehicle in the oncoming lane.

• We’ve also designed our software to reason about the context, which is essential for driving safely in busy cities. Our perception system lets our Driver know how to handle a pedestrian, a tree – and a pedestrian carrying a Christmas tree. If we pull up next to a bus by a crosswalk on Beach Street in Fisherman’s Wharf, our Driver can reason that hidden passengers may be getting off, and that they may soon cross the street.

• We’re also building greater flexibility into our driving software to handle unexpected changes to the road. If we’re driving on 19th Avenue during road work and our sensors spot traffic cones and road work signs, our perception system understands that they are guiding us out of the usual lane, and our planning and routing systems can automatically update the vehicle’s route to navigate the new layout.Making continual improvements to our technology

Making continual improvements to our technology

Our new partnership with Girl Scouts of Northeast Texas inspired us to ask if any of our Waymonauts are Girl Scout alums and how their experience influenced their journey to Waymo. With the organization’s focus on STEM, entrepreneurism, and leadership skills, it’s no surprise we found many alums among us, with really inspiring stories about what being a Girl Scout meant to them personally, how they found themselves in STEM roles, and how those two things are very closely intertwined. 

Emily Warman, Software Engineer, Planning/Behavior

5 years as a Girl Scout in Dupage County, IL

“To me, [being a] girl meant inside voices, dolls, clothes I couldn’t get dirty, legs without bruises, being small. I felt limited. I wanted to be seen for things I was good at: how fast I could run, how high I could climb, how brave I was… Girl Scouts was the first time I felt like I was a part of a group of girls. I didn’t have to be anything I wasn’t in Girl Scouts. It was the first time I felt like being a girl was OK, maybe even great.”

Maggie Graupera, Recruiter, Hardware Engineering

3 years as a Girl Scout in San Jose, CA

“I was a Girl Scout for several years. Cookie sales was a formative experience for me. The act of selling is a skill I started to harness at an early age through Girl Scouts. That experience prepared me in my career and ultimately led me to my role of selling the experience of working at Waymo (as a recruiter).”

Megan Quick, System Engineer

16 years as a Girl Scout and Girl Scout camp counselor in Colorado

“I think that some parts of the Girl Scout curriculum really helped with understanding STEM careers, and introducing STEM and engineering concepts. I became a mechanical engineer and that was influenced by many things, but the hands-on experiences that I got at the Girl Scout camps definitely gave me early experience with this. I also think that my leadership experience that I got in the Girl Scouts likely helped me to get a scholarship to college.”

Michelle Peacock, Global Head of Policy and Government Relations

Michelle’s daughter, Mary Charlotte, during her days as a Brownie

Sandy showing off her Waymo spirit!

Tell us a little bit about yourself, and how you first heard about Waymo.

I’m Xavier, and I live in Chandler, Arizona. I’ve been riding with Waymo for around a year now and first heard about Waymo when I saw them driving around my neighborhood. I started wondering what they were and finally looked it up and saw on the website that you could sign up to ride in them.

I’d heard about autonomous cars before on a few tech podcasts I listen to, but I didn’t know the technology was fully there yet or that there was a possibility of riding in one until I signed up. It took about a week to get accepted, and I’ve been riding since.

What do you normally use the service for?

I don’t own a car myself, so my primary mode of transportation is Waymo, other ride-hailing services, or the bus.

I mainly use Waymo to get to and from work. I’m a restaurant manager at Wendy’s and don’t live too far away; it’s a short 10 minute ride usually. I also use it to visit my mother every once in a while. 

We bet your mom loves that! What makes you choose Waymo over some of the other modes of transportation you mentioned?

Waymo generally is a nice ride, and you get a very consistent experience. The cars are always super clean and it feels futuristic, almost like you have a robot butler. You get in, press the button, and you’re on your way.

I compare it a lot to other ride-hailing services that aren’t automated. With those, you don’t know what you’re going to get. Sometimes the driver is having an off day, may speed, or not obey the traffic laws.

With Waymo, it picks the same route, goes the same speed, and feels more secure. I also like the fact that you get to specifically choose where it’s going to pick you up and drop you off.

Have your transportation habits changed at all through COVID-19?

I still feel comfortable using the service. I feel as comfortable as I do going to work, which I still have to get to somehow!

You’ve had the opportunity to ride a lot in our 100% fully autonomous cars with no human driver. Can you talk about what that’s like? Any major differences or similarities from the experience you’ve previously had with vehicle operators?

Every fully autonomous ride has been pleasant. For me personally, anytime I get in a car with a driver, Waymo or otherwise, I usually don’t play my music too loud or fully relax since I’m trying to consider the other person. But when I’m by myself, I’m more likely to change the temperature or adjust my seat and feel more at ease watching YouTube videos or listening to a podcast on my phone.

I also feel more comfortable changing the destination during the trip, like if I’m heading home from work and remember I need to stop by the store on the way home. It doesn’t feel like I’m inconveniencing anyone in fully autonomous rides.

That’s awesome to hear. Any feedback you have on the overall experience?

The only flaw I’ve noticed, but it’s not actually a flaw, is that sometimes Waymo is overly cautious. I can’t complain about that because it’s a good thing it’s cautious. Sometimes it will make decisions on the road, like routing, that don’t make sense to me all the time, but again, I understand it’s just being cautious.

In my head, all of these things are going to improve or already are.

What do you think the potential is for fully autonomous technology, both in times like these and in the future more broadly?

For me, I don’t have any doubt that autonomous cars are the way to go for the future, and if the world is just, it’s going to replace people driving cars. If it’s not necessary for humans to drive, why would we if it’s safer for computers to drive?

And, it’s going to have a good impact on the environment because if not everyone owns a car there’s going to be less cars out there because we’d all be sharing them.

Anything else you want to share about your experience?

I definitely recommend that people try it! It’s really fun and you feel like you’re part of the future.

Editor’s note: This blog was originally published by Let’s Talk Autonomous Driving on January 12, 2020.

Last year was a big one for us – we raised our first external investment round, we rolled out the 5th generation of our Waymo Driver and we opened Waymo One to the public, among other milestones. As we get started in the new year, we’ll have even more to share with you. And when we do, you’ll see us using more deliberate language, referring to our fully autonomous driving technology, and no longer referencing “self-driving.”

It may seem like a small change, but it’s an important one, because precision in language matters and could save lives. We’re hopeful that consistency will help differentiate the fully autonomous technology Waymo is developing from driver-assist technologies (sometimes erroneously referred to as “self-driving” technologies) that require oversight from licensed human drivers for safe operation. Regardless of who or what is at the helm, safely operating a vehicle on public roads requires careful execution of all the elements of the driving task. Today, the Waymo Driver makes billions of decisions each day as it safely moves people and goods to their destination in fully autonomous mode.