Page 23 - Read Online

Qi et al. Intell Robot 2021;1(1):18-57 Intelligence & Robotics
DOI: 10.20517/ir.2021.02

Research Article Open Access

Federated reinforcement learning: techniques, appli-
cations, and open challenges

2
1
1
Jiaju Qi , Qihao Zhou , Lei Lei , Kan Zheng 2
1 School of Engineering, University of Guelph, Guelph, ON N1G 2W1, Canada.
2 Intelligent Computing and Communications (IC ) Lab, Beijing University of Posts and Telecommunications, Beijing 100876, China.
2
Correspondence to: Dr. Lei Lei, School of Engineering, University of Guelph, 50 Stone Road East, Guelph, ON N1G 2W1, Canada.
E-mail: leil@uoguelph.ca

How to cite this article: Qi J, Zhou Q, Lei L, Zheng K. Federated reinforcement learning: techniques, applications, and open chal-
lenges. Intell Robot 2021;1(1):18-57. http://dx.doi.org/10.20517/ir.2021.02

Received: 24 Aug 2021 First Decision: 14 Sep 2021 Revised: 21 Sep 2021 Accepted: 22 Sep 2021 Published: 12 Oct 2021
Academic Editor: Simon X. Yang Copy Editor: Xi-Jun Chen Production Editor: Xi-Jun Chen

Abstract
This paper presents a comprehensive survey of federated reinforcement learning (FRL), an emerging and promising
field in reinforcement learning (RL). Starting with a tutorial of federated learning (FL) and RL, we then focus on the
introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance
of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL
algorithms can be divided into two categories, i.e., horizontal federated reinforcement learning and vertical federated
reinforcement learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the
evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition,
the existing works on FRL are summarized by application fields, including edge computing, communication, control
optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to
solving the open problems within FRL.

Keywords: Federated learning, reinforcement learning, federated reinforcement learning

1. INTRODUCTION
As machine learning (ML) develops, it becomes capable of solving increasingly complex problems, such as
image recognition, speech recognition, and semantic understanding. Despite the effectiveness of traditional
machine learning algorithms in several areas, the researchers found that scenes involving many parties are still

© The Author(s) 2021. Open Access This article is licensed under a Creative Commons Attribution 4.0
International License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, shar­
ing, adaptation, distribution and reproduction in any medium or format, for any purpose, even commercially, as long as you
give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate
if changes were made.

www.intellrobot.com