Page 47 - Read Online

Qi et al. Intell Robot 2021;1(1):18-57 I http://dx.doi.org/10.20517/ir.2021.02 Page 42

framework is presented for edge system [88] , named as “In-Edge AI”, to address optimization of mobile edge
computing, caching, and communication problems. The authors also propose some ideas and paradigms for
solving these problems by using DRL and Distributed DRL. To carry out dynamic system-level optimization
and reduce the unnecessary transmission load, “In-Edge AI” framework takes advantage of the collaboration
among edge nodes to exchange learning parameters for better training and inference of models. It has been
evaluated that the framework has high performance and relatively low learning overhead, while the mobile
communication system is cognitive and adaptive to the environment. The paper provides good prospects
for the application of FRL to edge computing, but there are still many challenges to overcome, including the
adaptive improvement of the algorithm, and the training time of the model from scratch etc.

Edge caching has been considered a promising technique for edge computing to meet the growing demands
for next-generation mobile networks and beyond. Addressing the adaptability and collaboration challenges of
the dynamic network environment, Wang et al. [89] proposes a device-to-device (D2D)-assisted heterogeneous
collaborative edge caching framework. User equipment (UE) in a mobile network uses the local DQN model
to make node selection and cache replacement decisions based on network status and historical information.
In other words, UE decides where to fetch content and which content should be replaced in its cache list. The
BS calculates aggregation weights based on the training evaluation indicators from UE. To solve the long-term
mixed-integer linear programming problem, the attention-weighted federated deep reinforcement learning
(AWFDRL) is presented, which optimizes the aggregation weights to avoid the imbalance of the local model
quality and improve the learning efficiency of the DQN. The convergence of the proposed algorithm is verified
and simulation results show that the AWFDRL framework can perform well on average delay, hit rate, and
offload traffic.

A federated solution for cooperative edge caching management in fog radio access networks (F-RANs) is pro-
posed [90] . Both edge computing and fog computing involve bringing intelligence and processing to the origins
of data. The key difference between the two architectures is where the computing node is positioned. A du-
eling deep Q-network based cooperative edge caching method is proposed to overcome the dimensionality
curse of RL problem and improve caching performance. Agents are developed in fog access points (F-APs)
and allowed to build a local caching model for optimal caching decisions based on the user content request
and the popularity of content. HFRL is applied to aggregate the local models into a global model in the cloud
server. The proposed method outperforms three classical content caching methods and two RL algorithms in
terms of reducing content request delays and increasing cache hit rates.

For edge-enabled IoT, Majidi et al. [91] proposes a dynamic cooperative caching method based on hierarchical
federated deep reinforcement learning (HFDRL), which is used to determine which content should be cached
or evicted by predicting future user requests. Edge devices that have a strong relationship are grouped into
a cluster and one head is selected for this cluster. The BS trains the Q-value based local model by using BS
states, content states, and request states. The head has enough processing and caching capabilities to deal with
model aggregation in the cluster. By categorizing edge devices hierarchically, HFDRL improves the response
time delay to keeps both small and large clusters from experiencing the disadvantages they could encounter.
Storage partitioning allows content to be stored in clusters at different levels using the storage space of each
device. The simulation results show the proposed method using MovieLens datasets improves the average
content access delay and the hit rate.

Considering the low latency requirements and privacy protection issue of IoV, the study of efficient and secure
caching methods has attracted many researchers. An FRL-empowered task caching problem with IoV has
been analyzed by Zhao et al. [92] . The work proposes a novel cooperative caching algorithm (CoCaRL) for
vehicular networks with multi-level FRL to dynamically determine which contents should be replaced and
where the content requests should be served. This paper develops a two-level aggregation mechanism for