Introduction to Markov Decision Processes

Author: Hu Qiying
Publisher:
Publish Date: 2000-07-01
Features: Markov Decision Processes (MDP) are theoretical tools for optimizing multi-stage decision-making processes in stochastic environments. Over the past few decades, with the emergence of numerous new models in fields such as ecological science, economic theory, communication engineering, and many others that require consideration of uncertainty and sequential decision-making, MDPs have seen significant theoretical and applied advancements. This book introduces the basic concepts, decision-making processes, and some fundamental theories of Markov Decision Processes starting with simple examples. It also covers various optimal criteria, including finite-stage criteria, discounted criteria, average criteria, weighted reward criteria, and probabilistic criteria. From a modeling perspective, it considers finite state spaces, countable state spaces, and general Borel state spaces; from the decision-making time perspective, it addresses discrete-time, continuous-time, and semi-Markov decision-making time problems. The book also introduces numerous application examples and modeling methods. This book can serve as a textbook for senior undergraduate and graduate students, as well as a reference for scholars and technicians in the fields of operations research, management science, information science, systems science, and computer science and engineering. The book provides generalized methods for handling three fundamental types of Markov Decision Process models: discrete-time, continuous-time, and semi-Markov. On this basis, the book explores generalized Markov Decision Processes such as partially observable states, multi-objective problems, constrained conditions, and Markov Decision Processes in stochastic changing environments.