Page 128 - Read Online

Boin et al. Intell Robot 2022;2(2):145­67 I http://dx.doi.org/10.20517/ir.2022.11 Page 153

Figure 3. Inter-FRL.

Figure 4. Intra-FRL.

In order to perform FRL, it has been proven that including an update delay between global FRL updates is
[5]
beneficial for performance . In addition, turning off FRL partway through training is important to allow
each agent to refine their models independently of each other such that they can perform best with respect
[5]
to their environments . Lastly, it has also been shown that global updates and local updates should not be
performed in the same episode [15] .

Two methods of aggregation are implemented in the system design, Inter-FRL (see Figure 3), and Intra-FRL
(see Figure 4). The proposed system is capable of aggregating both the model weights and gradients for each
model so that either type of parameter may be averaged for use in global updates. The FRL server has the
responsibility of averaging the parameters (model weights or gradients) across each agent in the system.

The pseudo-code for the Inter/Intra-FRL algorithm is presented in Algorithm 1. The system is designed to
allow the training of any number of equal length platoons. At the lowest level, a DDPG agent exists for each
vehicle in each platoon. As such, a DRL model must be initialized for each vehicle in the whole system. Each
DDPG agent trains separately from the others before data is uploaded to the FRL server. Federated averaging
is applied at a given time delay known as the FRL update delay, while being terminated at a given episode as