Course Overview

Graduate-level course covering modern RL techniques including offline RL, safe RL, and applications to real-world problems. Emphasis on bridging theory and practice with hands-on projects.

Course Description

This course provides an in-depth exploration of advanced reinforcement learning topics, with a focus on techniques and challenges relevant to real-world deployment. Students will gain both theoretical understanding and practical experience through implementation projects.

Learning Objectives

By the end of this course, students will be able to:

  1. Understand fundamental RL algorithms and their theoretical properties
  2. Implement and debug modern RL algorithms from scratch
  3. Identify appropriate RL formulations for real-world problems
  4. Recognize and address practical challenges in RL deployment
  5. Critically evaluate RL research papers and reproduce key results

Tentative Course Outline

Part I: Foundations (Weeks 1-4)

  • Markov Decision Processes and Dynamic Programming
  • Policy Gradient Methods
  • Value-Based Methods (Q-Learning, DQN)
  • Actor-Critic Algorithms

Part II: Advanced Methods (Weeks 5-8)

Offline Reinforcement Learning

  • Conservative Q-Learning
  • Behavior regularization
  • Dead-end identification

Safe Reinforcement Learning

  • Constrained MDPs
  • Risk-sensitive RL
  • Uncertainty quantification

Transfer Learning and Meta-Learning

  • Domain adaptation
  • Multi-task RL
  • Few-shot RL

Part III: Applications (Weeks 9-12)

  • RL for Language Models
  • Healthcare applications
  • Robotics and control
  • Real-world deployment considerations

Part IV: Research Projects (Weeks 13-16)

  • Student-led research projects
  • Paper presentations
  • Final project presentations


Prerequisites

Required:

  • Strong programming skills (Python)
  • Probability and statistics
  • Linear algebra
  • Basic machine learning (equivalent to an introductory ML course)

Recommended:

  • Prior exposure to deep learning


Course Format

  • Lectures: 2x weekly, covering theory and algorithms
  • Lab Sessions: Weekly hands-on implementation exercises
  • Paper Discussions: Student-led presentations of recent papers
  • Projects: Individual or team projects applying RL to real problems


Assessment

  • Homework assignments (30%): Implementation-focused problem sets
  • Paper presentations (20%): Present and lead discussion on research papers
  • Midterm exam (20%): Covering fundamental concepts
  • Final project (30%): Original research or substantial application


Textbooks

  • Sutton & Barto: “Reinforcement Learning: An Introduction” (2nd ed)
  • Bertsekas: “Dynamic Programming and Optimal Control”

Papers & Surveys

  • Curated reading list of foundational and recent papers
  • Survey papers on offline RL, safe RL, and applications

Practical Resources

  • OpenAI Gym / Gymnasium
  • Stable-Baselines3
  • D4RL benchmark suite


Coming Soon

Detailed syllabus, lecture notes, and course materials will be made available as the course develops. Stay tuned!