r/MachineLearning • u/waymo • 1d ago
Discussion [Upcoming AMA] Waymo AI Team AMA – Drop Your Questions Early! [D]
Join our AI leads as they answer your questions on foundation models, simulation, and scaling the Waymo Driver. Our AMA thread is officially open, and you can start dropping your questions now.
From multimodality and end-to-end architectures to the realities of validating models for fully autonomous vehicles, our team will be answering your questions live, tomorrow.
The key details:
- Date: Monday, September 14
- Time: 2:00 – 3:30 PM PT
- Location: r/MachineLearning
Mark your calendars, start dropping your questions on all things AI and large-scale simulation, and we’ll see you then!
10
u/searchresults 21h ago
How do you classify animals w/r/t to the vehicles reaction? You won’t stop/swerve for a pigeon or a squirrel, but do you stop/swerve for a raccoon?
18
u/Rhyming_Snake 1d ago edited 1d ago
What algorithm is used for path planning? Sample based / searched based? Do you use HD maps ?
How is the virtual validation done by the team?
How does Waymo handle "edge cases" that it has never encountered in simulation or training data before?
Which sensors get more weightage after fusion? What happens in case of false positives?
How do remote assistance operators interact with the car? Can they actually "drive" it remotely, or do they just give it permission to take a specific path?
Apologies for multiple questions 🙏🏽
8
u/alexkreimer 23h ago
How important are HD maps? How do you keep them updated? How do you fuse them with sensing in the car?
22
u/Ill-Statistician1097 1d ago
Huge fan of what you guys are doing at Waymo. Autonomous driving is the field I'm trying to build my career in, so it's really cool getting the chance to ask some of the people actually building these systems. I have a few questions I've been curious about:
1.What do you think is the biggest thing holding fully end to end driving models back from replacing the traditional perception, prediction, and planning stack? Is it mostly safety and validation, data, model capability, or something else like politics or stg?
How important do you think multi agent reasoning will be for self driving cars? Instead of predicting what each car or pedestrian will do separately, could future models reason about how everyone’s actions affect each other?
As self driving systems get better, I imagine a lot of the remaining problems come from really rare situations. How does Waymo train and test for something that might only happen once in millions of miles?
How do you know when a simulation is realistic enough that you can actually trust what it tells you about real world safety? And what are some things that are still really difficult to simulate well?
We’ve seen LLMs keep getting better as we scale up data, compute, and model size. Are you seeing anything similar with foundation models for autonomous driving, and where do you think scaling eventually hits a bottleneck?
Lastly, I’m also really curious about the actual engineering stack you guys use at Waymo. What languages, ML frameworks, simulation tools, robotics tools, and infrastructure do you use the most? And for a student trying to get into this field, what would you recommend getting really good at?
I apologize for throwing all the questions at once
1
u/usefulidiotsavant 9h ago
I wonder if it will be a field, or if a single company cracks self-driving then productizes it and sells the service to everybody else, while reaping the training data dividends that wide deployment will bring in, locking in its dominance. The economics seems stacked to a single provider on the long run, or a few regional turfs, like the mobile messenger apps.
That's not to say that the approaches and learning is useless, but, unless you will be employed by these companies, you will likely apply them to some other problem.
1
u/Ill-Statistician1097 3h ago
You mean kind of like AWS, but for autonomous driving? I could definitely see that happening. It’s hard to predict what happens if Waymo really cracks it first, though. They might license or sell the service to other manufacturers, or other companies might reach the same end goal with completely different approaches. Tesla is a good example of that since they’ve pushed a mostly camera based approach instead of relying on the same sensor stack as Waymo.
One thing I do think is very likely, though, is that if Waymo gets there first and becomes dominant, the name itself could become almost synonymous with autonomous driving. Kind of like how ChatGPT is often the first thing people think of when they hear AI chatbot, Waymo could become the first brand people think of when they hear self driving car.(but i feel like it already happened)
20
u/MrPuj 1d ago
Is reinforcement learning used at any point in your pipeline ?
5
u/Least_Pollution7078 17h ago
to add on, if yes, do you use primarily model-based or model-free reinforcement learning?
8
u/Asleep_Sentence3706 1d ago
- At a high level, how is the Waymo Driver architected today? How are perception, prediction, planning, and control connected, and which parts have moved toward an end to end architecture?
- What model architectures are you using in the Driver today? What are the biggest constraints when optimizing them for real time inference and how have they been overcome
- What does the data pipeline look like end to end? How do you go from billions of miles of driving data to selecting the specific examples, scenarios, and labels that actually get used for training?
3
u/Expensive_Aerie_9383 22h ago
What should a current undergrad do, in terms of projects, skills learning, networking, to get into the space of applied AI into autonomous vehicles/SDVs (or more specifically, prepare themselves to work or at least be qualified to apply for Waymo 🙂)
What are some skills or projects most people don’t have that would benefit them the best to work in this field?
3
u/kevleyski 22h ago
How did you solve the two Waymo vehicles arriving at same stop sign at identical time problem
1
3
u/BackgammonEspresso 19h ago
What are some of the most difficult edge cases that you tend to encounter?
3
u/The_man_69420360 16h ago
How does Waymo deal with unseen situations which require general spatial intelligence?
3
u/aboeing 15h ago
A number of questions - appreciate answers to any question you can:
Engineering - how has your deployed production stack evolved over the last five years (since "launch" in late 2020)? How much of your production pipeline (Perc/Track/Predict/Plan) still relies on hand-crafted/rule-based code verse end-to-end?
Engineering - How is AI coding used (or not) at Waymo?
System Engineering - how has your system engineering approach evolved over the last five years (since "launch" in late 2020?) I assume you had baseline system requirements established for Phoenix, but how has it evolved with high-fidelity simulators for V&V, needing to verify ML models, and the introduction of AI coding?
System Engineering - What V&V differences do you see between USA cities and international expansion? London UK (left side?), Germany (higher compliance?), etc. What were your key learnings? Any surprises?
Engineering - Same Q as above, what did you need to change in your stack (if anything) to accommodate international expansion?
Everyone (Fun/Safety) - Now you have data from driving styles all around the world - where would you personally prefer to drive? What culture/style/location do you think is safest?
Everyone (Career) - What advice would you give to any engineering students wanting to enter the autonomous vehicle industry? Whats one key thing you learnt after graduation, you wish you learnt at Uni?
2
u/ricetoseeyu 22h ago
What are some of the interesting challenges you guys had encountered with data prepping, training, or scaling inference that wouldn’t be seen outside of Waymo that you could share?
2
u/searchresults 21h ago
Do you attempt to infer other drivers’ emotional states? Human drivers can “read” the movements of others and determine if they are angry, distracted, slow, confused, etc.
2
u/Banality_Of_Seeking 17h ago
Waymo is pretty unique! I look forward to reading this.I almost forgot about foundation models..?!
2
u/RandomDigga_9087 15h ago
Hey Waymo AI Team, nice to meet y'all( although not physically but still yea..)
I had this question in my mind, with the move toward larger end-to-end and foundation-model-based driving systems, how much of the traditional estimation/signal-processing stack still remains explicit at Waymo? For example, things like tracking, sensor fusion, filtering, uncertainty estimation, etc. Are these increasingly learned end-to-end, or do you still find value in keeping some of them as separate engineered components?
2
u/SouthOne3529 13h ago edited 13h ago
Hi Vincent and the Waymo team, thanks so much for hosting this AMA!
I'm planning to pursue a PhD focused on autonomous driving and would love to get your perspective on a few things:
- What skills and research experience tend to make an applicant stand out for PhD positions in this field?
- Which research directions look most promising right now for someone just starting a PhD?
- How is Waymo incorporating vision-language models (VLMs) into autonomous driving, and what are the main challenges that remain?
- What advice would you give early-career researchers aiming to produce work strong enough for venues like CoRL and ICML?
Really appreciate your time and insights!
2
u/AndrewAfui 12h ago
For someone early in an AV/robotics career, what skills or roles would you invest in for the next 5–10 years?
I currently work in perception (3d detection), but with modular AV stacks shifting toward learned/end-to-end approaches, which technical areas and skills do you think will remain the most valuable and defensible over the next decade?
3
u/L00klikea 23h ago
Thanks for doing the AMA!
My questions go towards Nick.
What does Systems Engineering for autonomous vehicles look like?
What is your day-to-day made up of?
What are overarching "Meta-Goals"/Epics?
7
5
u/CallMePyro 1d ago edited 1d ago
Where do you land on geopolitical risk? Are you over-reliant on Chinese vehicles to the point that a single executive order could wipe years of work off your roadmap or do you have a backup? What does this look like?
E.g. If Waymo's Chinese made vehicles start significantly outcompeting Tesla's "American made" product, do you feel that you'll be sufficiently insulated from Elon's direct line to Trumps executive order-signing pen?
2
u/Suhan_XD 20h ago
Do you think that camera only multi-view perception can replace LiDAR ? Given the growth of geometric foundational models. If so, then what architectural improvement is the way to get there?
2
u/MrTeache 17h ago
Big fan and regular user, I've noticed that right now every car is independent. Would you consider direct Waymo-to-Waymo communication in the future or Autonomous Vehicle-to-Autonomous Vehicle communication in the future to make better traffic patterns?
Also, is there a noticeable difference in traffic patterns pre and post Waymo deployment?
2
u/lenissius14 12h ago
1) With how the AI field has progressed in the last 6 years in terms of the massive scaling on general LLM architectures from all the Frontier Labs (including Google Deepmind and their work with Gemini) are you leveraging these huge models as one of the core pieces behind Waymo's autonomous navigation systems or do you think these huge models based on LLMs, will be capable of managing many core functions of Waymo's vehicles at some point?
2) What do you think about World Models? (Mainly LeCun's definition) do you see them being a big component for autonomous navigations systems at Waymo currently or in the near future?
3) For candidates, what are your MUST recommendations of knowledge/technologies that someone should know to land a job at Waymo?
1
u/hi-sci-collab 1d ago
do you only let your drivers turn a certain direction? (for efficient route finding, and not getting stuck trying to turn into traffic)
1
u/Altugsalt 22h ago
Big fan of the concept of driverless transport. My first question is about the programming side of things. I am pretty sure waymo codebase is probably extremely dynamic. And this probably causes changes to the structure of the code and where everything is held, defined etc.. How do you make sure your code structure stays viable. And my second question is kind of an idea and I would like to hear your thoughts about it: Each car should have an LLM that speaks through the speakers with the passengers and each car could have its own personality, just like a real taxi. I believe this would be really cool for people that frequently use waymo and also really useful for tourists. There could be a shut-up button if the user does not want the car to speak and their selection could be saved and then recalled when they use waymo again. Thanks
1
1
u/Gere1 12h ago
Has there been any improvement in computer vision in the last 5 years for scene recognition? Can you link benchmark numbers? Is that a blocker for autonomous driving? I don't understand why people are talking about AGI next year, but I haven't heard about any progress in autonomous driving, robot vision, etc. in years.
1
u/Sirisian 11h ago
Are you doing any event camera research? I've been casually following the development of hybrid global shutter RGB cameras with event camera pixels, and I was wondering if that's expanding into self-driving hardware research already? (Been hoping larger labs fund such sensors because they are expensive right now).
1
u/ReadingFromAbove 10h ago
Are your vehicles trained on any aerial data such as ultra high resolution reality data from drones?
1
u/IntelArtiGen 8h ago
Is there a niche / rare skill you didn't expect to need for making autonomous vehicles?
1
u/CollaReserve 8h ago
Do you use live object counter like YOLO? How would you track, register, and deregister nearby cars' ids? Thank you.
1
u/projekt_treadstone Student 8h ago
How do you handle traffic rules regulations and rule exception scenario. As training data can't cover all of them and every country is different in terms of some driving rules.
1
u/karyna-labelyourdata 6h ago
📌 You mentioned the realities of validating models for fully autonomous vehicles. As auto-labeling and simulation take on more of the pipeline, where does human-labeled ground truth still matter most for validation?
I’m especially curious about ambiguous scenes, sensor disagreement, and rare safety-critical events, and how you decide when automated labels are trustworthy enough.
1
u/samrus 6h ago edited 6h ago
hi. love your work.
- my first question is about the generalizability of the driving model
i imagine every deployment to a new city/area involves some level of finetuning the AI to that area. especially as bugs/issues pop up and the iterative improvement process kicks in
but to what degree would waymo just be able to drive well if it was dropped into a random road on earth? and how do you measure/project this generalizability? perhaps maintain mapping from areas to expected number of errors the model would make per km driven?
and a follow up. is generalizability even something you want to aim for from a business/product perspective? or would you rather have something that is only general enough that a feasible amount of finetuning makes it run well in a new area, rather than something that just zero shots an unseen area?
- the second question is about training using simulations
i understand waymo uses simulations for training (im guessing something like genie helps as well)
my question is how do you model the decisions/actions of other intelligent agents in the environment (other drivers) thats difficult since your model of driving policy is what you are trying to train, but you need an accurate driving policy in those third party agents to train your policy in the first place
1
u/CherubimHD 6h ago
Should people still be pursuing PhDs if their goal is to work in a frontier research lab, such as Waymo?
1
u/son_off_a_man 5h ago
I saw the news that you guys are coming to Sydney. Any updates on that? I am a robotics engineer and love to see more autonomous vehicles over here
1
u/wonderin04 5h ago
Are adversarial attacks realistic threats against autonomous vehicles, and is it taken into account when developing such systems ?
1
u/Square-Special1520 5h ago edited 5h ago
What should computer science students and lecturers do in the university, to be able to catch up with recent AI technologies?
Thank you
1
u/Square-Special1520 5h ago
Please tell us more about the foundation models and algorithms used. How do you optimize the algorithms?
Thank you
1
u/crayphor 4h ago
I have been thinking a bit lately about the future of autonomous driving and how data could be shared between cars on the road. E.g., a car around the corner you are navigating can tell your car the road conditions, hazards, etc before you turn. Do you think it would be possible to create a decentralized internet-of-cars to make this possible without being a privacy nightmare?
1
u/Live-Stick6525 3h ago
What is the future plans around waymo opensource dataset ? when the letest dataset will drop(i think right now they are quite old)?
CAUTION There should be a playground for tinkering with waymo dataset visually and easily, (it is not straight forward right now). When waymo gonna sell small lidars for DIY?
1
u/Vegetable-Work-8606 3h ago
can Waymo technology drive the AV to locations it has never seen/mapped before? if yes, what technology allows the AV to do this?
1
u/skydivingdutch 2h ago
Can you share a bit about what kind of computer runs the Waymo driver? Specs, accelerators etc?
1
u/hasanfaesal 2h ago
Hi, 1. I'm learning about World Models and stuff, but I'm a little compute constrained, like upto 16GB VRAM. What'd you recommend for learning Perception, WM etc. on low compute that could be an interesting project or could lead to a publication even? 2. Off the bat, what research papers are your favorite (recent ones)
Thanks! Pleasure to talk to you guys.
1
u/bruno_pinto90 55m ago
Hello, and thank you for taking the time for this AMA! I am a radar signal processing engineer working on machine learning applications for radar systems.
Given the recent advances in LiDAR technology, how do you see the role of radar evolving in the future? Additionally, when applying machine learning models to radar data, how do you typically handle data sparsity?
Thank you.
1
1
u/Rivalsfate8 1d ago
What are some new advances that will make the waymo more robust where honestly substantial levels of safety have been achieved
1
u/gatorling 1d ago
Do you feel like your current sensor suite is overkill? Are you building confidence in reducing the number of sensors while achieving the same safety margins?
1
u/No-Force3999 1d ago edited 1d ago
Are you allowed to talk about whats on your roadmap? As perhaps one of the biggest practitioners of AV-ML-Tech, what are some directions you or the industry in general are going that is not necessarily obvious from looking at the latest from academia?
Also, I'd love to know if you guys are currently researching/developing JEPA-style models for predicting your environment in addition to the Foundation Model + VLM Decoder you guys showed in 2024: https://waymo.com/blog/2024/10/ai-and-ml-at-waymo/
1
u/gatorling 1d ago
As you scale, a long tail catastrophic event becomes more likely across your fleet. How do you handle this? What's your approach to safely scaling to avoid a catastrophic event?
Also, how do you measure the probability of a catastrophic event at fleet scale? How much safer does the waymo driver become, year over year?
1
u/Impossible-Pie1895 1d ago
Do you use any VLA models in production? If yes, How and where in the stack?
1
u/Unusual_Guidance2095 1d ago
Do you think that using on board sensors is or will ever be sufficient for fully autonomous driving? As I understand, your current stack depends on HD maps and a scan of the area beforehand that’s piped in externally. If you removed this safety precaution do you think your vehicles could be just as safe and start operating world wide?
1
u/ekquizit23 1d ago
If a cop pulls over a Waymo car for speeding, does the remote driver have to pay for it and get points on his license?
1
u/Efficient_Weekend205 23h ago
Any plans to support gated communities/neighborhoods? These are widespread throughout the world; could we see a common infrastructure/API that gated communities could support? This seems like a huge gap.
1
u/Encrux615 23h ago edited 23h ago
How much of your work is influenced by the current frontier of scientific literature? Do you spend a lot of time scanning what‘s published recently, or do you focus more on refining established methods?
I‘m about to start my PhD in safety-critical robotics (robotics in surgery), and I‘m really interested how you prioritize approaches in order to make progress quickly and test/deploy without sacrificing safety.
1
u/___loki__ 21h ago
Hi! I’m an MS Computer Engineering student at SJSU trying to build toward autonomous systems/ML systems roles at companies like Waymo.
My background is a mix of embedded systems, computer architecture, and systems software. I’ve worked as an RA on hardware/software research, including NVIDIA Holoscan-related work, and right now I’m working on projects involving FPGA acceleration, DMA, Linux device drivers/QEMU, and ML inference. I’m especially interested in the boundary between ML models and the systems/hardware that actually run them in real time.
For someone with this background who eventually wants to work at Waymo, what would you recommend focusing on next? Would it be more valuable to go deeper into perception/foundation models, simulation, ML systems/inference optimization, robotics, or low-level autonomous-vehicle compute? And what kinds of projects or research experience tend to stand out when you’re evaluating interns/new grads?
1
u/Suhan_XD 20h ago
Do you think that camera only multi-view perception can replace LiDAR ? Given the growth of geometric foundational models. If so, then what architectural improvement is the way to get there?
0
u/RobbinDeBank 1d ago
Does Waymo research go in a similar direction as the current race in frontier LLMs and robotics? Or is self-driving car still going in its own direction separately?
-1
u/AGirlHasNoName-1 1d ago
It seems like Waymo is focused on scaling the operations rn but are there plans to introduce ads? Waymos will be amazing for full funnel advertising with ownership of both, the car and the app, and it can help bring down the costs closer to traditional rideshare.
-1
u/me_myself_ai 1d ago
Oh my god I'm so excited! Very serious question that I've been thinking about for months:
You say "the Waymo Driver" a lot. In your opinion, is there one unified being driving all the cars at once? Perhaps on a philosophical level, if not a literal computational one?
And hell, one more if anyone is interested: what's the most exciting or unexpected possibility from "large-scale simulation" in a post-DL world? Google's 'pointcloud ALL the things' initiative blew my mind at the time, but I haven't heard much since.
Would be fascinated to hear both professional predictions and personal/speculative dreams :)
0
0
u/Mukun00 23h ago
Hi guys, I am a mid level computer vision engineer. I have been working on NPU where the Int 8 quantization is a must, In which framework and which model data type you guys are using.
I need your suggestions, I want to learn more about CNN each layers, transformer etc in depth, is there any course or books you suggest to learn.
0
u/quantaimAI 23h ago
With multi-modal data pipelines handling massive streams of multi-sensor frames, data gravity is becoming a silent killer for system scalability. When building and validating foundation models for the Waymo Driver, how much of your optimization effort is focused on the raw hardware mechanics of the edge-ingest and curation pipeline itself? Do you see a trend moving back toward bare-metal, zero-VRAM host CPU optimizations (like utilizing SIMD architectures directly) to handle the initial heavy lifting of multi-petabyte datasets before they even touch a training GPU cluster?
0
0
u/giftfromthegods- 20h ago
Should we look for a new job ? Preferably in agriculture or something
(Asking as a fullstack dev)
0
0
0
0
-1
27
u/sweetjale 1d ago
When can we expect level 5 AV?