PPO implementation for a 3D CartPole balancing task using PyBullet. The agent learns to balance a pole upright on a cart by applying horizontal forces. Uses a custom Gymnasium environment with PyBullet physics simulation.

Challenge: Stable balancing was never achieved. Can you tune the hyperparameters, reward function, or environment to get stable pole balancing?

Note: On Windows, WSL is recommended (unknown if it works natively on Windows).

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -r requirements.txt

Training:

python train.py

View training progress: tensorboard --logdir logs/tensorboard

Evaluation:

python evaluate.py

Keyboard controls: r (reset), l (reload model), t (toggle auto-reset), y (select model)

Configuration can be modified in config.py. Trained models are saved to models/.

• cartpole_3d_env.py: Custom Gymnasium environment implementing the 3D CartPole physics simulation using PyBullet.
• cartpole_3d/: Contains URDF files and meshes for the 3D physics model. See cartpole_3d/README.md for URDF modification notes.