"Dimitri P. Bertseka,美國MIT終身教授，美國國家工程院院士，清華大學復雜與網絡化繫統研究中心客座教授,電氣工程與計算機科學領域國際知名作者，著有《非線性規劃》《網絡優化》《凸優化》等十幾本暢銷教材和專著。本書的目的是考慮大型且具有挑戰性的多階段決策問題，這些問題原則上可以通過動態規劃和很優控制來解決，但它們的準確解決方案在計算上是難以處理的。本書討論依賴於近似的解決方法，以產生具有足夠性能的次優策略。這些方法統稱為增強學習，也可以叫做近似動態規劃和神經動態規劃等。本書的等

●1. Exact Dynamic Programming
1.1. Deterministic Dynamic Programming
1.1.1. Deterministic Problems
1.1.2. The Dynamic Programming Algorithm
1.1.3. Approximation in Value Space
1.2. Stochastic Dynamic Programming
1.3. Examples, Variations, and Simplifications
1.3.1. Deterministic Shortest Path Problems
1.3.2. Discrete Deterministic Optimization
1.3.3. Problems with a Termination State
1.3.4. Forecasts
1.3.5. Problems with Uncontrollable State Components
1.3.6. Partial State Information and Belief States
1.3.7. Linear Quadratic Optimal Control
1.3.8. Systems with Unknown Parameters - Adaptive Control
1.4. Reinforcement Learning and Optimal Control - Some Terminology
1.5. Notes and Sources
2. Approximation in Value Space
2.1. Approximation Approaches in Reinforcement Learning
2.1.1. General Issues of Approximation in Value Space
2.1.2. Off-Line and On-Line Methods
2.1.3. Model-Based Simplification of the Lookahead Minimization
2.1.4. Model-Free off-Line Q-Factor Approximation
2.1.5. Approximation in Policy Space on Top of Approximation in Value Space
2.1.6. When is Approximation in Value Space Effective？
2.2. ltistep Lookahead
2.2.1. ltistep Lookahead and Rolling Horizon
2.2.2. ltistep Lookahead and Deterministic Problems
2.3. Problem Approximation
2.3.1. Enforced Decomposition
2.3.2. Probabilistic Approximation - Certainty Equivalent Control
2.4. Rollout and the Policy Improvement Principle
2.4.1. On-Line Rollout for Deterministic Discrete Optimization
2.4.2. Stochastic Rollout and Monte Carlo Tree Search
2.4.3. Rollout with an Expert
2.5. On-Line Rollout for Deterministic Infinite-Spaces Problems Optimization Heuristics
2.5.1. Model Predictive Control
2.5.2. Target Tubes and the Constrained Controllability Condition
2.5.3. Variants of Model Predictive Control
2.6. Notes and Sources
3. Parametric Approximation
3.1. Approximation Architectures
3.1.1. Linear and Nonlinear Feature-Based Architectures
3.1.2. Training of Linear and Nonlinear Architectures
3.1.3. Incremental Gradient and Newton Methods
3.2. Neural Networks
3.2.1. Training of Neural Networks
3.2.2. ltilayer and Deep Neural Networks
3.3. Sequential Dynamic Programming Approximation
3.4. Q-Factor Parametric Approximation
3.5. Parametric Approximation in Policy Space by Classification
3.6. Notes and Sources
4. Infinite Horizon Dynamic Programming
4.1. An Overview of Infinite Horizon Problems
4.2. Stochastic Shortest Path Problems
4.3. Discounted Problems
4.4. Semi-Markov Discounted Problems
4.5. Asynchronous Distributed Value Iteration
4.6. Policy Iteration
4.6.1. Exact Policy Iteration
4.6.2. Optimistic and ltistep Lookahead Policy Iteration
4.6.3. Policy Iteration for Q-factors
4.7. Notes and Sources
4.8. Appendix: Mathematical Analysis
4.8.1. Proofs for Stochastic Shortest Path Problems
4.8.2. Proofs for Discounted Problems
4.8.3. Convergence of Exact and Optimistic Policy Iteration
5. Infinite Horizon Reinforcement Learning
5.1. Approximation in Value Space - Performance Bounds
5.1.1. Limited Lookahead
5.1.2. Rollout and Approximate Policy Improvement
5.1.3. Approximate Policy Iteration
5.2. Fitted Value Iteration
5.3. Simulation-Based Policy Iteration with Parametric Approximation
5.3.1. Self-Learning and Actor-Critic Methods
5.3.2. Model-Based Variant of a Critic-Only Method
5.3.3. Model-Free Variant of a Critic-Only Method
5.3.4. Implementation Issues of Parametric Policy Iteration
5.3.5. Convergence Issues of Parametric Policy Iteration Oscillations
5.4. Q-Learning
5.4.1. Optimistic Policy Iteration with Parametric Q-Factor Approximation - SARSA and DQN
5.5. Additional Methods - Temporal Differences
……

本書的主要內容包括：第1章動態規劃的準確求解；第2章值空間的逼近；第3章參數逼近；第4章無限時間動態規劃；第5章無限時間強化學習；第6章集結技術。通過本書讀者可以較為全面地了解動態規劃、近似動態規劃和強化學習的理論框架、主流算法的工作原理和近期新發展。本書可用作人工智能或繫統與控制科學等相關專業的高年級本科生或研究生的教材，也適合開展相關研究工作的專業技術人員作為參考用書。

(美)德梅萃·P.博塞卡斯 著