3 posts tagged with "webots"

📌 Key Takeaways (TL;DR)

• Problem: When a goal lies directly behind an obstacle, the myopic nature of distance-delta rewards (+0.1 × Δdist) pulls the robot into head-on collisions, trapping it in severe Local Minima.
• Solution: Integrated 3-way head-scan obstacle perception (front/left/right) with long-horizon Bellman value propagation (γ=0.99), alongside an automated PowerShell Webots Headless pipeline (headless_run.ps1) executing multi-seed training across seeds 0, 1, and 2.
• Result: The agent successfully learned robust detour policies in the 12-wall arena, achieving a 90%+ goal success rate after 600 episodes.

📌 Key Takeaways (TL;DR)

• Problem: Sparse reward setups (+1.0 at goal, -1.0 at collision) cause severe exploration delays, causing the agent to wander or spin for thousands of episodes before stumbling upon the target.
• Solution: Implemented a multi-objective Dense Reward Shaping mechanism combining potential-based Euclidean distance delta rewards (+0.1 × Δdist) with step (-0.01), turning (-0.01), and backward (-0.05) penalties, paired with exponential epsilon decay (1.0 → 0.05).
• Result: The moving average reward shifted into a steady positive regime (0.7+) within 500 episodes in the no_obstacle scenario, achieving a 98%+ goal success rate.

📌 Key Takeaways (TL;DR)

• Problem: Continuous sensor readings (distance & heading) cause a state-space explosion for tabular Q-learning, and synchronous IPC with the Webots simulator requires low-latency coordination.
• Solution: Designed a geometrically partitioned 5-dimensional continuous sensor mapper producing 3 × 8 × 3 × 3 × 3 = 648 discrete states (discretizer.py) and a line-delimited JSON IPC socket pipeline over TCP port 5000.
• Result: Achieved O(1) fast Q-Table indexing and reliable 32ms physics synchronization between the simulator and Python agent.