Page 82 - Read Online

Zhang et al. Intell Robot 2022;2(3):275­97 Intelligence & Robotics
DOI: 10.20517/ir.2022.20

Review Open Access

Deep reinforcement learning for real-world quadrupedal

locomotion: a comprehensive review

Hongyin Zhang, Li He, Donglin Wang
School of Engineering, Westlake University, Hangzhou 310000, Zhejiang, China.

Correspondence to: Dr. Donglin Wang, School of Engineering, Westlake University, Dunyu Road No.600, Xihu District, Hangzhou
310000, Zhejiang, China. E-mail: wangdonglin@westlake.edu.cn; ORCID: 0000-0002-8188-3735
How to cite this article: Zhang H, He L, Wang D. Deep reinforcement learning for real-world quadrupedal locomotion: a compre-
hensive review. Intell Robot 2022;2(3):275-97. http://dx.doi.org/10.20517/ir.2022.20

Received: 30 Jun 2022 First Decision: 25 Jul 2022 Revised: 27 Jul 2022 Accepted: 22 Aug 2022 Published: 1 Sep 2022

Academic Editor: Simon X. Yang Copy Editor: Jia-Xin Zhang Production Editor: Jia-Xin Zhang

Abstract
Building controllers for legged robots with agility and intelligence has been one of the typical challenges in the pursuit
of artificial intelligence (AI). As an important part of the AI field, deep reinforcement learning (DRL) can realize se-
quential decision making without physical modeling through end-to-end learning and has achieved a series of major
breakthroughs in quadrupedal locomotion research. In this review article, we systematically organize and summarize
relevant important literature, covering DRL algorithms from problem setting to advanced learning methods. These
algorithms alleviate the specific problems encountered in the practical application of robots to a certain extent. We
first elaborate on the general development trend in this field from several aspects, such as the DRL algorithms, simu-
lation environments, and hardware platforms. Moreover, core components in the algorithm design, such as state and
action spaces, reward functions, and solutions to reality gap problems, are highlighted and summarized. We further
discuss open problems and propose promising future research directions to discover new areas of research.

Keywords: Deep reinforcement learning, quadrupedal locomotion, reality gap

1. INTRODUCTION
Wheeledandtrackedrobotsarestillunabletonavigatethemostchallengingterraininthenaturalenvironment,
and their stability may be severely compromised. Quadrupedal locomotion, on the other hand, can greatly ex-
pand the agility of robot behavior, as legged robots can choose safe and stable footholds within their kinematic

© The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0
International License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, shar­
ing, adaptation, distribution and reproduction in any medium or format, for any purpose, even commercially, as long as you
give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate
if changes were made.

www.intellrobot.com