Skip to content

About

Deep Reinforcement Learning from mathematical foundations to PyTorch implementations: rigorous proofs, step-by-step derivations of objectives and gradient estimators, and reproducible experiments on Gymnasium and MuJoCo benchmarks spanning policy gradients, actor-critic methods, value-based learning, and continuous control.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Deep Reinforcement Learning

Python 3.12+ PyTorch Gymnasium Development License: MIT

A PyTorch reinforcement learning library in which every algorithm starts as a mathematical derivation and ends as readable, tested code you can train with.

  • Derivation first. Each algorithm has a written derivation, and the code follows it: what is stored, where gradients stop, what stays fixed during an update.
  • Readable. The core modules explain, step by step, why each step exists.
  • Practical. Checkpoints and resume, saved policies that load without any training code, and observation normalization saved with the policy.

Active development: APIs may change. The examples are runnable recipes, not reproduced benchmarks.

Algorithms

Algorithm Code Derivation Status
One-step actor-critic one_step_actor_critic.py Actor-Critic, with a baseline ✅
A2C (GAE) a2c.py GAE Actor-Critic ✅
Natural Policy Gradient npg.py NPG ✅
TRPO trpo.py TRPO ✅
PPO-Clip ppo.py PPO ✅
DQN dqn.py DQN ✅
Double DQN dqn.py (double_q=True) Double DQN ✅
DDPG — DPG, DDPG derivation done

A2C, NPG, TRPO, and PPO share the on-policy trainer; DQN and Double DQN use the off-policy trainer. One-step actor-critic is a standalone teaching implementation.

Quickstart

Requires Python 3.12+ and uv.

git clone https://github.com/SaiSampathKedari/Deep-Reinforcement-Learning.git
cd Deep-Reinforcement-Learning
uv sync --extra tb --extra video

Train PPO on CartPole-v1 (CPU, 8 environments, 102,400 steps):

uv run python -m examples.gymnasium.cartpole.run_cartpole_ppo \
  --output-dir outputs/gymnasium/cartpole/ppo/seed_0/run_01

Evaluate the saved policy, and view the training curves:

uv run python -m examples.gymnasium.cartpole.evaluate_cartpole \
  outputs/gymnasium/cartpole/ppo/seed_0/run_01/policy --episodes 10

uv run tensorboard --logdir outputs/gymnasium/cartpole/ppo/seed_0/run_01/events

Each run writes one folder (use a new one per run):

run_01/
├── config.json      every setting of the run
├── metrics.jsonl    episodes, training reports, evaluations
├── events/          TensorBoard logs
├── checkpoints/     last.pt and periodic step_<N>.pt, for resuming
└── policy/          the trained policy, loadable without any training code

The CartPole guide covers video recording, rendering, and resuming from a checkpoint.

Examples

Task What it shows
CartPole-v1 Discrete actions; the CPU starting point
InvertedPendulum-v5 A Gaussian policy for continuous control
Hopper-v5 Continuous-action locomotion
Humanoid-v5 High-dimensional control with observation normalization

Each example keeps its networks, settings, training script, and evaluator together, and calls the shared library. The MuJoCo tasks need one more extra:

uv sync --extra mujoco --extra tb --extra video

How the Library Is Organized

Part Files Responsibility
Learners algorithms/ Configs, targets, losses, and parameter updates
Trainers train_on_policy.py, train_off_policy.py When to collect, learn, evaluate, and save
Data collectors.py, buffers.py, advantages.py Environment stepping, rollout/replay storage, GAE
Networks and processing policies.py, value.py, processors.py, inference.py Wrappers for your networks; observation and action processing
Evaluation and logging evaluate.py, logger.py Episode returns, JSONL metrics, TensorBoard
Saving and loading checkpoint.py, policy_io.py Training checkpoints and exported policies

Buffers store raw observations; normalization is applied when a network reads them, with statistics the trainer updates on a fixed schedule. Network architectures stay with each example. The CartPole training script shows the full assembly: environments, networks, learner, logger, evaluator, and the training call.

Checkpoints and Saved Policies

  • Checkpoints (checkpoints/) hold everything needed to continue training: networks, optimizers, observation statistics, progress, and random-number state. Pass --resume-from .../checkpoints/last.pt to continue a run. A resumed run starts fresh episodes, and off-policy runs refill an empty replay buffer before learning again.
  • Policies (policy/) hold only what acting needs: a manifest, the model weights, and the observation statistics. They are written when training finishes, and load without a learner or optimizer.

To evaluate a checkpoint before training finishes, export it to a new folder:

from deeprl.policy_io import export_policy_from_checkpoint

export_policy_from_checkpoint(
    "outputs/gymnasium/cartpole/ppo/seed_0/run_01/checkpoints/last.pt",
    "outputs/gymnasium/cartpole/ppo/seed_0/run_01/policy_snapshot",
)

The full rules are documented in checkpoint.py and policy_io.py.

From Derivations to Code

The derivations are working specifications. They decide what the buffers keep, where gradients stop, and which quantities stay fixed during an update: for example, termination-aware GAE, PPO's fixed old log-probabilities, and TRPO's rollback. The DQN derivation splits Q-learning into three independently scheduled steps (data collection, target-network refresh, and Q-function fitting), which is exactly how the off-policy trainer and DQN learner are built.

Generalized Q-learning: data collection, target-network refresh, and Q-function fitting

Browse all derivations. Their numbering continues from the Reinforcement Learning repository: 13–20 cover policy gradients and actor-critic; 21–27 cover NPG, TRPO, PPO, DQN, Double DQN, deterministic policy gradients, and DDPG.

Roadmap

  • Done: on-policy methods (actor-critic, A2C, NPG, TRPO, PPO), DQN, and Double DQN.
  • Next: DDPG, TD3, SAC, and multi-seed Gymnasium and MuJoCo benchmarks.
  • Later: offline RL, imitation learning, model-based RL and world models, robotics integration, and VLA post-training.

Related Repositories

Contact

License

Released under the MIT License.

About

Deep Reinforcement Learning from mathematical foundations to PyTorch implementations: rigorous proofs, step-by-step derivations of objectives and gradient estimators, and reproducible experiments on Gymnasium and MuJoCo benchmarks spanning policy gradients, actor-critic methods, value-based learning, and continuous control.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages