One post tagged with "Danny Driess"

In our previous lecture on Autonomous Systems & Reliability, we explored how software agents operate within digital execution sandboxes, governed by multi-perimeter trust boundaries and AI Control protocols. Yet, a fundamental question confronts the artificial intelligence community:

"Can an intelligence confined solely to computer screens, generating text tokens and compiling software scripts, ever attain true Artificial General Intelligence (AGI)? Our physical reality is not governed by virtual shell commands; it is shaped by gravity, friction, 3D Euclidean geometry, and the delicate contact dynamics of rigid and deformable objects."

What happens when we bestow a foundation model with physical embodiment — providing it with two high-resolution RGB eyes (cameras) and a seven-degree-of-freedom steel arm (robotic manipulator)?

• Picking up a fragile raw egg without fracturing the shell and placing it onto a frying pan;
• Sorting children's scattered toys by color into designated storage containers across a cluttered living room;
• Aligning and inserting micro-electronic connectors with sub-millimeter precision on an industrial assembly line.

Welcome to Lecture 16 of Stanford CS329A: Self-Improving AI Agents, the penultimate lecture of Part 4: Frontiers (Frontiers in Reasoning & Autonomy). Grounded in the pioneering research of Dr. Danny Driess (Founding Research Scientist at Physical Intelligence, former Lead at Google DeepMind Robotics), we investigate the architectural revolution of Vision-Language-Action (VLA) Models:

• Actions as Language Tokens (RT-2, DeepMind 2023): Treating continuous 7DoF robot arm kinematics as foreign language tokens within a 55B parameter vision-language model.
• The Open-Source Vanguard (OpenVLA, 2024): Defeating 55B closed models with a 7B parameter architecture via SigLIP + DINOv2 dual-vision fusion, quantile binning, and LoRA adaptation running on consumer gaming GPUs.
• Continuous Flow Matching & Diffusion Policies ($\pi_0$): Overcoming discrete token jitter to master high-frequency deformable object manipulation, such as folding laundry and assembling boxes.
• Embodied Safety Architectures: Multi-tier software action clipping coupled with hardware force-torque sensor reflex loops.