Reinforcement learning for autonomous navigation.

Behavior starts with the environment contract

My graduate research explored reinforcement learning for autonomous navigation without treating the simulator as a neutral backdrop. Vehicle dynamics, obstacle placement, observations, reward terms, contacts, and termination rules jointly define what a policy can learn. I built a custom Ackermann-drive environment in MuJoCo through dm_control, then used it to train and inspect PPO and SAC agents with Stable-Baselines3.

The goal was to evaluate safety, stability, and generalization across obstacle layouts rather than only maximize episodic return.

My role

I owned the research implementation loop: translating the navigation problem into an RL environment, modeling the car and its interactions, defining observations and actions, wiring training through Stable-Baselines3, and debugging the gap between reward curves and visible behavior.

• Built and iterated the Ackermann-drive MuJoCo model and dm_control task.
• Implemented obstacle generation, contact-based collision checks, resets, and episode termination.
• Designed reward terms and inspected how their relative scale changed policy behavior.
• Configured PPO and SAC training and compared policy behavior under a shared environment contract.
• Explored CNN, graph-based, and transformer-style representations for spatial observations.
• Produced trajectory and representation artifacts for qualitative debugging alongside scalar logs.

System design

1. Simulation and dynamics: MuJoCo resolves vehicle motion, steering behavior, geometry, and contacts for an Ackermann-drive model.
2. Task layer: dm_control manages observations, action application, randomized obstacle scenes, reset logic, reward calculation, and termination.
3. Representation: proprioceptive state and spatial observations are converted into policy inputs; learned encoder experiments included visual and graph-oriented feature extraction.
4. Policy learning: Stable-Baselines3 provides PPO and SAC training loops while custom PyTorch modules handle representation experiments.
5. Inspection: episode logs, contact checks, trajectories, and rendered behavior are used together to find disagreement between numerical reward and navigation quality.

MuJoCo dynamics + obstacles
          ↓
dm_control observation / reward / termination
          ↓
state or learned spatial representation
          ↓
PPO or SAC policy
          ↓
steering + drive action
          ↓
contacts, trajectory, clearance, return

What I built

Vehicle and action model

The simulated vehicle used Ackermann steering rather than a point-mass shortcut. The action interface exposed steering and drive commands, while the environment translated them into actuator controls and enforced bounds that kept the learning problem tied to the modeled vehicle.

Scene generation and collision logic

Obstacle placement and reset logic created variation across episodes. Collision checks used MuJoCo contact data and named geometry pairs, allowing termination and penalties to follow physical interactions instead of relying only on center-point distance.

Reward and termination

Reward design combined progress and motion incentives with safety costs. Collision, boundary behavior, control quality, and episode termination were treated as coupled decisions: changing one can create a shortcut elsewhere, so the behavior was inspected after each revision rather than judged from return alone.

Representation experiments

The image shown on this research page is a representation-work artifact from experiments with structured spatial input; it is not a screenshot of the MuJoCo vehicle. CNN-based visual features and graph-oriented alternatives were explored to test how the policy could consume richer environmental structure beyond low-dimensional state alone.

Evaluation approach

The intended evaluation separates learning progress from navigation quality. PPO and SAC should be compared only when the dynamics, observation contract, reward, training budget, and held-out layouts are controlled.

• Repeatability: log environment, library, model, and training configuration with explicit random seeds.
• Generalization: reserve obstacle layouts and seed ranges that the policy does not see during training.
• Safety: measure collisions and minimum clearance, not only aggregate episode return.
• Task behavior: track episode duration, forward progress, stalled motion, and boundary violations.
• Control quality: inspect steering oscillation and abrupt command changes alongside task completion.
• Qualitative audit: retain representative and worst-case rollouts so failure modes remain visible.

These are the criteria for the refresh, not published results. Metric values will be added only after the rerun can be reproduced from a documented configuration.

Observed failure modes

The implementation was checked for the following failure modes:

• Reward sensitivity: progress incentives can dominate collision costs, so the visible path must be checked whenever reward weights change.
• Contact ambiguity: incorrect geometry names or contact-pair handling can let real collisions pass uncounted—or terminate on harmless contacts.
• Layout memorization: a narrow obstacle generator can produce a convincing policy that only understands a recurring scene pattern.
• Representation instability: a more complex encoder can improve the training curve without yielding consistently safer motion on held-out layouts.
• Seed dependence: evaluation should cover a fixed set of seeds and adverse cases rather than a single successful rollout.

Current limitations

• The work remains simulation research and has not been deployed on a physical vehicle.
• A final, versioned benchmark table is not yet available after the interruption in active development.
• A one-command training package and downloadable checkpoint are not currently available.
• Generalization across broader sensing conditions and dynamics variation still needs controlled testing.
• ONNX and TensorRT conversion paths were explored; the models were not deployed in production.

Rebuild the baseline, then add complexity

1. Freeze a minimal state-based baseline with pinned dependencies and a deterministic smoke test.
2. Publish the environment contract, configuration files, and distinct train/evaluation seed lists.
3. Rerun PPO and SAC under matched budgets; report central tendency and spread, not a best seed.
4. Release representative, median, and failure rollouts with collision and clearance summaries.
5. Reintroduce richer learned representations only after the baseline can be reproduced.
6. Add dynamics and observation randomization before evaluating transfer beyond simulation.