This work was conducted by researchers João Victor Feio Gonçalves (UFPA), John Sousa (UFPA), Rafael Veiga (UFPA), Lucas de Lima Bastos (UNIFESSPA), Lucas de Sousa Pacheco (UFPA), Iago Medeiros (UFPA), Denis Rosário (UFPA), and Eduardo Coelho Cerqueira (UFPA).
They developed ASTRA, a new method designed to optimize Federated Learning in scenarios where data across devices is highly diverse and disconnected. The core of the innovation lies in Dual-Space Regularization, which combines physical constraints on model weights with semantic self-distillation to ensure that local predictions remain aligned with the global objective.
To enable use on resource-constrained devices, the authors implement Curriculum-Based Scaling, activating the “instructor” model intensively only at the beginning and subsequently on a periodic basis. The purpose of the study is to mitigate client drift and prevent catastrophic system divergence, achieving higher accuracy than traditional methods with only a small increase in computational cost.
Client drift
In edge networks, data exhibits significant asymmetry across different devices (non-IID data). The impact on local training occurs because clients optimize exclusively based on their own private data, and local weight updates diverge significantly from the global objective. The consequences for FL are as follows:
■ Catastrophic divergence: the average of conflicting gradients destroys the model’s overall performance
■ Convergence failures: standard methods (such as FedAvg) exhibit severe instability.
The goal, then, is to keep local updates semantically aligned with the global consensus without suppressing local learning,
In this way, the authors demonstrate that it is possible to reconcile structural stability and logical flexibility to create more robust and efficient decentralized artificial intelligence models.
Teacher-Student Distillation
Teacher-Student Distillation, in the context of sources, is a semantic approach that aims to align a model’s outputs (logits) rather than just its weights.
The central concept is based on the transfer of “dark knowledge” from a robust “Teacher” model to a local “Student” model. The Mechanism—the transfer occurs via the output logits (pre-softmax activations)—seeks to minimize the Kullback-Leibler (KL) divergence between the student’s and teacher’s probability distributions.
Semantic consistency, which is reinforced in this process, is a technique that ensures the local model’s decision boundaries remain aligned with the global consensus, allowing the model to retain its classification logic even as the physical weights change to adapt to local data.
A temperature parameter is used to “smooth” the probability distributions; this amplifies the importance of low-probability classes, facilitating the transfer of relationships between classes (the “dark knowledge”) rather than just rigid labels.
ASTRA uses a variant called Self-Distillation, in which the periodically frozen global model acts as the Teacher and the local model under training acts as the Student.
Mentorship Phase
After the initial model stabilization stage, the ASTRA method enters a new training phase: the Mentorship Phase. The idea is that, instead of having the global model guide the local models throughout the entire process, the system begins to make periodic interventions, allowing the devices to operate more autonomously.
At the beginning of training, during the so-called Foundation Phase, the global model—the “Teacher”—continuously guides the local models, the “Students.” In the Mentorship Phase, this relationship changes. The Teacher begins to monitor learning at set intervals, acting as a sort of supervisor who intervenes only when necessary.
With each cycle of training rounds, the Teacher is activated to perform a check-in. At this point, the local model receives a semantic correction that helps keep its predictions aligned with the system’s overall objective. Between these interventions, however, the Teacher remains inactive.
It is during this interval that the so-called Autonomous Mode comes into play. Without the need to execute forward passes from the global model to perform distillation, the local model can focus on its own data and specialize according to the characteristics of that environment.
The strategy also has a direct impact on the use of computational resources. By reducing the frequency with which the global model needs to participate in training, ASTRA decreases VRAM consumption and the number of floating-point operations (FLOPs)—an aspect that is particularly relevant for edge devices, which typically operate with more limited computational resources.
Computational Challenge
A critical limitation of standard distillation is the computational cost, as it requires a forward pass from the teacher at each training step. ASTRA addresses this by activating the teacher’s signal only periodically, according to a curriculum schedule, to conserve the resources of edge devices. The global model is corrected from time to time. It is not possible to have the mentoring process active all the time; however, the goal is to have an efficient solution in which the global model does not need to be constantly providing guidance; things happen autonomously, and the model corrects what is necessary on an as-needed basis.