MAQDRL: QuadTree-Based Multi-Agent Federated Deep Q-Learning for Collision Avoidance and Routing Optimization with MEC

Abstract Autonomous vehicle routing in complex urban environments suffers from high computational overheads, energy inefficiency, and increased collision risks using traditional routing algorithms. These limitations necessitate innovative approaches to address scalable and real-time decision-making requirements for multi-agent systems. This research proposes a reinforcement learning-based Multi-agent Federated Deep Q-Network (MAQDRL) model, which integrates federated learning with QuadTree spatial partitioning and indexing framework for collision detection. The model employs a decentralized training approach to optimize routing efficiency and reduce energy consumption. Each agent of MAQDRL trains the Deep Q-Network (DQN) independently using local state information. Model parameters are periodically synchronized through federated learning to align policies while preserving data privacy. The QuadTree partitions the environment dynamically based on agent density, focusing computational resources on high-interaction areas to ensure efficient collision detection and proximity querying. Advanced preprocessing techniques like state normalization, action encoding, and reward scaling stabilize training and enhance policy convergence. The framework further employs an adaptive exploration-exploitation strategy and decentralized decision-making, allowing agents to collaboratively achieve optimized routing while mitigating collisions and reducing energy consumption in complex, high-density scenarios. Experimental results demonstrate that the proposed MAQDRL is a promising solution for advancing autonomous vehicle routing systems. MAQDRL achieves a 97.8% success rate, outperforming existing research - MARC (92.2%), MAPPO (94.8%), and MADDPG (95.2%).MAQDRL reduces collision frequency by 73.08% over MAPPO and 66.13% over MADDPG, while energy consumption is lowered by 56.52% ov er MAPPO and 55.98% over MADDPG on a CVRPTW-derived spatial navigation benchmark. Introduction As autonomous vehicles become integral to smart transportation systems, ensuring safe, scalable, and efficient navigation in complex urban environments remains a major challenge. Real-time routing and collision avoidance require rapid decisions based on high-frequency sensor inputs, yet this sense–plan–act loop faces two major bottlenecks. First, transmitting raw sensor streams such as LiDAR and V2X to centralized servers introduces latency that can exceed sub-100 ms safety thresholds and may violate regional data-privacy regulations. Second, collision detection in dense multi-agent environments typically relies on exhaustive pairwise checks, resulting in \(\mathcal {O}(N^2)\) computational cost as the number of agents grows. Edge computing has emerged as a promising paradigm to address these limitations by shifting computation closer to data sources and enabling local inference on roadside MEC units. This reduces end-to-end latency, alleviates bandwidth pressure, and keeps sensitive sensor data local. However, most reinforcement-learning (RL) controllers remain centrally trained and periodically synchronize model weights with edge devices, which introduces communication overhead and limits adaptability in non-stationary environments. Moreover, existing collision-avoidance strategies often depend on hand-crafted heuristics that fail in dense or rapidly changing traffic. Recent deployments further highlight the need for tighter integration between edge computing and learning-based control. Surveys of federated learning at the edge emphasize that privacy-preserving model updates and on-device inference can substantially reduce bandwidth consumption while maintaining data sovereignty [4].Likewise, cloud–edge–end architectures such as Fed4UL demonstrate the benefits of decentralized learning under non-IID data distributions, underscoring the relevance of federated approaches for autonomous mobility systems [30]. Parallel advances in Multi-Agent Reinforcement Learning (MARL), including MAPPO, MADDPG, and MARC [10,11,12], have achieved strong performance on benchmarks such as the Capacitated Vehicle Routing Problem with Time Windows (CVRPTW) [6]. Nonetheless, their dependence on centralized critics, joint action spaces, or global replay buffers introduces scalability challenges and unstable convergence in crowded environments. Furthermore, these methods lack a spatial index for efficient collision checking, limiting their applicability in real-world high-density scenarios. To overcome these limitations, we propose the Multi-Agent Federated Deep Q Reinforcement Learning (MAQDRL) framework, which integrates federated learning with dynamic QuadTree-based spatial partitioning. Each agent trains a lightweight Deep Q-Network (DQN) using only local observations, while federated averaging [7] periodically aligns model parameters without sharing raw data. In parallel, a QuadTree structure [5] organizes the environment according to agent density, enabling efficient \(\mathcal {O}(\log N)\) spatial queries and substantially reducing the cost of collision detection. Our key contributions are as follows. First, we introduce MAQDRL, a Federated DQN enhanced with QuadTree indexing for decentralized collision-aware path planning. Second, the algorithm achieves scalable, privacy-preserving training and logarithmic-time spatial queries, aligning naturally with edge computing constraints. Third, experiments on a CVRPTW-derived spatial navigation setting show that MAQDRL surpasses MAPPO, MADDPG, and MARC in task success, energy efficiency, and collision avoidance. The evaluation focuses on decentralized collision-aware navigation rather than full CVRPTW route feasibility. Literature Review Existing Models Overview The Multi-Agent Deep Deterministic Policy Gradient MADDPG algorithm builds upon DDPG foundations by giving each agent its own critical assessment system, which makes evaluations by adjusting to the strategies and actions produced by other agents. Through this method, agents become efficient at displaying cooperative as well as competitive behaviors. However, the scalability of MADDPG becomes a problem because each critic needs to handle the complete set of actions available to all participating agents, according to [10]. MAPPO represents an improvement on PPO by implementing stable operation to create a version suitable for multi-agent scenarios. While MAPPO performs stable updates to its policies, its performance degrades in active competition among agents who pursue different objectives or interact in dynamic environments [11]. MARC (Multi-Agent Reinforcement Collaboration) pursues joint cooperation through combined rewards and unified action spaces that develop collective agent teamwork. However, the cooperative task strengths of MARC become weaknesses when agents need unique policies, as it produces suboptimal results and leads to inferior performance during specialized strategy execution [12]. Recent Developments in Multi-Agent Systems [15] used decentralized deep reinforcement learning as an approach to avoid collisions within dynamic areas where pedestrians were present. Their scalable method shows limitations when adjusting to various environmental conditions since it reduces reliability in unpredictable but uncontrolled environments. [16] developed a UAV framework utilizing DRL alongside edge computing for reducing UAV decision delays therefore enabling immediate aerial system processing. The flexible nature of DRL needs additional innovation for improving multi-agent coordination during unpredictable circumstances. [17] employed a sequential deterministic policy gradient gradient to address navigation challenges in UAVs, proving effective in unknown environments. The study presents a robust model but could be extended to explore its applicability in more densely populated or varied environments. [18] focused on federated deep reinforcement learning for vehicle control. They successfully demonstrated enhanced data privacy and system scalability; however, the integration of these systems into broader, more heterogeneous networks remains a challenge. Recent studies have further highlighted the role of federated learning in vehicular and intelligent edge environments. [37] proposed a federated generative adversarial network-based approach for heterogeneous vehicle scheduling in the Internet of Vehicles (IoV), demonstrating how collaborative model training can support vehicle scheduling without directly sharing raw vehicular data. [38] introduced a personalized asynchronous federated learning framework for intelligent vehicular computing, addressing system heterogeneity, delayed client participation, and personalization in dynamic vehicular networks. In addition, [39] developed a personalized federated meta-learning method with a constrained hypernetwork to improve learning under non-IID data distributions. These studies are closely related to the federated-learning aspect of MAQDRL because they address privacy preservation, heterogeneity, personalization, and non-IID data challenges in distributed intelligent systems. However, their primary focus is on scheduling, personalized aggregation, or meta-learning. In contrast, the proposed MAQDRL framework integrates federated Deep Q-learning with QuadTree-based spatial indexing to support decentralized collision-aware routing and efficient local neighbourhood querying in multi-agent navigation environments. Recent advances have further refined the integration of federated learning with spatial optimization for improved efficiency. [35] proposed TinyFDRL for energy-efficient trajectory planning in space-air-ground networks using lightweight federated DRL models, achieving 23% energy reduction while preserving data privacy. Similarly, [36] introduced a centralized control framework for multiple UAVs that combines DRL with QuadTree spatial partitioning, demonstrating 35% faster convergence and 28% lower collision rates compared to decentralized approaches. [20] applies federated reinforcement learning in a decentralized UAV swarm navigation problem, using a DRL framework to handle path planning and collision avoidance. The environment simulates multiple UAVs and ground stations. Results show improved navigation efficiency and collision reduction (15%-20% fewer collisions) compared to centralized learning. While ensuring data privacy and scalability, limitations include handling non-non-independent and identically distributed data and communication overhead. [2] introduces a multi-agent DRL framework to plan paths for UAV swarms in dynamic, obstacle-rich environments. It leverages local state observations and communication among agents. Experimental results show a 20% improvement in collision avoidance and route efficiency over baseline algorithms. Advantages include scalability and robustness, but complexity increases with larger swarm sizes. [22] proposes an adaptive multi-agent DQN approach for UAV swarms operating in dynamic environments with moving obstacles. The UAVs learn collision-free navigation strategies based on local observations. The method improves collision avoidance success rates by 18% and ensures timely route adjustments. Limitations involve handling extremely dense environments and ensuring fast convergence. [23] proposes a multi-agent deep reinforcement learning framework for coordinated UAV routing and path planning in dynamic, city-like environments. Using simulated terrains and varying user demands, the approach enhances navigation efficiency and cuts collision rates by about 20%. It leverages scalable coordination and adapts to uncertain conditions, but increased computational overhead and longer convergence times limit its large-scale deployment. [24] utilizes deep reinforcement learning for cooperative path planning in a team of UAVs navigating uncertain, obstacle-rich terrains. Simulated complex environments and varying wind conditions serve as the dataset. The approach achieves up to 15% improved route efficiency and lower collision rates. Advantages include robust adaptation to dynamic conditions, but higher computational overhead and longer training times remain limitations. [25] integrates federated learning with multi-agent RL to enable UAV swarms to learn collision avoidance policies without sharing raw observations. The scenario includes multiple UAVs and dynamic obstacles in simulated datasets. Results show improved safety margins and energy savings. While privacy-preserving and scalable, it may face communication overhead and slower convergence when scaling to large swarms. [26] presents a cooperative multi-agent DRL framework for UAV navigation in changing wireless network conditions. Agents operate on simulated city maps with real-world traffic patterns, achieving reduced collisions and more stable connectivity ( 10–12% improvement). Its main strengths are adaptability to varying environments and communication patterns, though training complexity and parameter tuning pose challenges. [27] leverages deep reinforcement learning to coordinate multiple UAVs for safe and efficient mission execution. The simulated scenarios feature dynamic targets, obstacles, and time-varying flight conditions. The method achieves a reduction in collision incidence and improved response times of about 15%. While robust in moderately complex environments, scaling to very large formations and handling extremely dense obstacle fields remain areas for improvement. Limitations of Existing Methods The common limitations across these existing methodologies include: - Scalability: Increased computational overhead and performance degradation in larger, more complex environments. - Agent Coordination: Difficulty in maintaining optimal cooperation in competitive settings, especially when agents have conflicting goals. - Complexity Handling: Inefficiencies in environments with dynamic inter-agent interactions and changing operational contexts. Research Gaps Existing centralized reinforcement learning approaches are plagued by numerous research limitations that constrain their performance in multi-agent environments. Scalability is a primary concern, with existing approaches depending on mounting computational loads and diminishing performance in larger, more complex environments. One primary inefficiency is the need for each agent to compare with every other agent, with exponentially growing computational complexity and power demand. This large-scale comparison not only slows down decision-making but also precludes real-time operation. Additionally, existing models are deficient in proper agent coordination, particularly in competitive environments where agents have different objectives, and are susceptible to inducing more collisions. Existing approaches also do not have the capability to optimize energy usage and path efficiency, leading to wasteful routing and inefficient resource utilization. Lastly, centralized approaches are not responsive in dynamic environments where inter-agent interactions and operating conditions are constantly evolving, and therefore it is difficult to sustain optimal performance. Another extreme limitation is the inability to create personalized reward structures, with existing frameworks having a tendency to use a one-size-fits-all approach for all agents, constraining their ability to personalize rewards per individual objectives. Bridging these gaps is critical to improving collision avoidance, energy efficiency, path optimization, and overall system performance in large-scale multi-agent systems.Table 1 presents a comparison of representative centralized MARL, federated MARL, and the proposed MAQDRL approach. Federated DQN is included as a federated baseline to clarify the distinction between federated parameter sharing and the additional QuadTree-based spatial indexing used in MAQDRL. Motivation and Introduction to Anticipated Methodology The challenges in existing multi-agent reinforcement learning approaches inspired the creation of the Multi-Agent Federated Deep Q-Network (MAQDRL). Leveraging a federated learning framework, MAQDRL facilitates the development of a shared policy while allowing agents to retain unique adaptations, improving scalability and minimizing computational demands. The inclusion of a QuadTree structure enhances collision detection efficiency in dynamic and complex environments. This decentralized methodology strikes a balance between cooperative and competitive interactions, promoting both autonomy and effective collaboration. The proposed MAQDRL framework is introduced in Section 1, highlighting its motivation and contributions. Section 2 reviews supporting research, comparing MAQDRL with existing models like MADDPG and MAPPO. Section 3 explains the System Model, detailing the structured grid environment, agent dynamics, and QuadTree-based spatial partitioning. Materials and Method This detailing the core components of the proposed MAQDRL framework. It includes the System Model, describing the spatial environment and QuadTree-based partitioning; Environment Description, outlining agent dynamics and constraints; Task Allocation, which explains how agents are assigned goals; and Collision Avoidance, detailing the QuadTree’s role in preventing conflicts. Additionally, Federated Knowledge Sharing covers decentralized learning, while Neural Network Architecture presents the model design. The MAQDRL Methodology outlines the overall approach and framework integration. Finally, the Proposed MAQDRL Algorithm elaborates on the reinforcement learning methodology used for multi-agent decision-making. System Model The system model for the Federated Multi-Agent Deep Q-Learning (MAQDRL) framework operates within a bounded two-dimensional grid environment \(\mathcal {G} \subset \mathbb {R}^2\), partitioned into a \(100 \times 100\) spatial grid. Each agent is assigned a unique goal location. To navigate efficiently while minimizing computational complexity, the grid is dynamically subdivided using a QuadTree structure \(\mathcal {T}\). This partitioning allows for localized collision detection with logarithmic computational overhead, as depicted in Fig. 1. At the start of each episode, all N agents are randomly initialized at distinct grid coordinates \((x^i_0,y^i_0)\in [0,100]\times [0,100]\), with no two agents sharing the same cell. After initialization, agents move deterministically according to their learned DQN policy: at each timestep, an agent selects one of the discrete actions (up, down, left, right, stay) that maximizes its local Q-value. During training we use an \(\epsilon\)-greedy strategy for exploration, but once deployed the agent’s movement follows the greedy policy without random drift. Dynamic Spatial Partitioning To optimize collision detection and reduce computational complexity, the grid \(\mathcal {G}\) is recursively subdivided using a QuadTree structure \(\mathcal {T}\), as defined in (1): where \(Q \subset \mathcal {G}\) represents a region within the grid, |Q| is the number of agents in region Q, C is the agent capacity per quadrant, and \(Q_i\) are the four sub-regions resulting from partitioning Q. This recursive subdivision continues until all regions contain a manageable number of agents, allowing for efficient localized collision detection with logarithmic computational overhead. Dynamic Subdivision and Merge Policy The QuadTree structure is dynamically updated at each timestep. After agents move, the tree is cleared and rebuilt to reflect their new positions. Each insertion or query operation remains efficient at \(\mathcal {O}(\log N)\) due to the balanced tree structure. A region is subdivided into four child quadrants if it exceeds a preset capacity C, ensuring fine-grained spatial resolution in densely populated zones. To avoid excessive fragmentation, a lazy merge policy is adopted: sibling nodes whose combined agent count falls below C/2 are merged back into their parent region. This keeps the structure compact and adaptive, preserving speed even with modest agent counts. The adaptivity of the tree enables efficient collision queries in dense clusters while maintaining simplicity in sparse areas, making it well-suited to dynamic multi-agent systems. As shown in Fig. 1, the QuadTree partitions are visualized, illustrating how the grid is adaptively segmented based on agent movements and density. Grid Boundaries Agent positions are restricted to \((x, y) \in [0, 100] \times [0, 100]\). The boundary conditions, as stated in (2), ensure: where \(a_i^t\) and \(b_i^t\) denote the (latitude, longitude) X and Y coordinates of agent \(a_i\) at time \(t\), and \(T\) is the maximum episode length. This constraint prevents agents from moving outside the defined grid, maintaining the integrity of the environment. Edge Computing Integration MAQDRL is motivated by MEC-assisted edge-computing environments, where communication bandwidth, model size, and server-side aggregation cost are important constraints. In the proposed framework, each agent trains a lightweight local DQN policy using its own observations, while only model parameters are periodically exchanged with the MEC-side federated aggregator. This avoids transmission of raw trajectories, sensor observations, or replay-buffer samples. The QuadTree module further reduces the cost of neighbourhood querying by limiting collision checks to spatially relevant local regions. Therefore, the edge relevance of MAQDRL is analyzed in this work from the perspective of communication overhead, model compactness, and MEC-side aggregation complexity. Direct edge-device measurements such as inference latency, memory footprint, and energy consumption on constrained hardware are considered an important direction for future validation (Table 2). Agent Definition In the MAQDRL framework, each agent \(a_i\) is described by a state vector \(s_i^t = (x_i^t, y_i^t, t, d_{\text {goal}}^i)\), where: - \((x_i^t, y_i^t)\) is the agent’s current position at time \(t\), - \(t\) is the current timestep within the episode (\(t \in [0, T]\)), - \(d_{\text {goal}}^i\) is the Euclidean distance to the agent’s goal, calculated in (3): this distance metric plays a crucial role in decision-making and reward assessments. Each agent selects actions from the discrete set \(\mathcal {A}_i = \{\text {up}, \text {down}, \text {left}, \text {right}, \text {stay}\}\), with movement updates determined by the displacement vector \(\Delta a\). The agent’s position at the next timestep is updated in (4): this deterministic movement ensures that the agent will consistently move toward its goal once trained, without any random drift. The state vector \(s_i^t = (x_i^t, y_i^t, t, d_{\text {goal}}^i)\) captures the essential information about the agent’s position, elapsed time, and goal distance. These features are critical for guiding the agent’s decision-making process. The agent selects actions based on a policy learned via deep Q-learning. To ensure stable training, techniques like state normalization and action encoding (Section 3.6.3) are applied, enabling consistent and unbiased decision-making across features and actions. MAQDRL Architecture Overview To overcome the limitations of traditional centralized reinforcement learning in dynamic, multi-agent environments, we propose the Multi-Agent Federated Deep Q-Reinforcement Learning (MAQDRL) framework. This subsection provides a unified explanation of the key architectural components and how they extend the conventional Deep Q-Network (DQN) paradigm to address decentralization, coordination, and scalability challenges. 1. Multi-Agent Extension of DQN In a standard DQN [8], a single agent learns an action-value function \(Q(s, a; \phi )\) by interacting with a stationary environment. In MAQDRL, we extend this concept to \(N\) independent agents, each maintaining its own local policy \(Q_i(\textbf{s}_i^t, \textbf{a}_i^t; \phi _i)\) based on its partial observation \(\textbf{s}_i^t\). This decentralized setup allows agents to learn asynchronously from their own experiences. Local replay buffers and independent target networks help stabilize learning despite inter-agent non-stationarity. 2. Federated Learning Module To coordinate learning across agents while preserving data privacy, MAQDRL adopts a federated learning scheme [7]. After a fixed number of episodes, each agent shares its model parameters \(\phi _i\) (but not raw data) with a central aggregator. The global model \(\phi\) is computed via federated averaging: This synchronized model is redistributed to agents to update their local networks. The cycle promotes shared knowledge while maintaining decentralized execution and reduces communication overhead compared to full trajectory sharing. 3. QuadTree-Based Spatial Indexing A critical innovation in MAQDRL is the integration of a dynamic QuadTree [5] to accelerate collision detection and reduce spatial query complexity. The 2D grid environment is recursively subdivided based on agent density. Each agent inserts its current position into the QuadTree structure, which enables logarithmic-time retrieval of nearby agents. During action selection, this allows agents to query only local neighborhoods rather than compute costly pairwise distances across all agents. This reduces the per-timestep complexity of collision detection from \(\mathcal {O}(N^2)\) to \(\mathcal {O}(\log N)\), making MAQDRL scalable to dense urban settings. MAQDRL systematically combines agent-local DQNs, periodic federated aggregation, and spatially adaptive QuadTree indexing. Together, these modules enable scalable, decentralized, and collision-aware navigation in high-density multi-agent environments. Task Allocation and Collision Avoidance Collision Boundary Here \(\Delta c\) and \(\Delta d\) are predefined proximity thresholds. This boundary defines a region around the agent where a collision is considered imminent if another agent enters it. Referring to Fig. 1, the collision boundaries are visually represented as areas surrounding each agent, highlighting potential collision zones. Each agent maintains a collision boundary \(B_i^t\) at time \(t\), as defined in (5): : Collision Detection The QuadTree \(\mathcal {T}\) facilitates efficient collision detection by querying agents within \(B_i^t\). A collision indicator \(\delta _{\text {collision}}^i\) is defined in (6): this ensures that collisions are detected efficiently using spatial partitioning techniques. This indicator checks whether the collision boundary of agent \(a_i\) overlaps with that of any other agent \(a_j\). If an overlap exists, indicating a potential collision, \(\delta _{\text {collision}}^i\) is set to 1. Utilizing the QuadTree structure allows for efficient detection of such overlaps, especially in densely populated regions of the grid. In Fig. 1, the collision detection process is conceptually shown, illustrating how the QuadTree aids in identifying agents within proximity. Federated Knowledge Sharing Federated sharing of data in this case allows agents to learn together to enhance navigation policies while maintaining data privacy. Parameters of the model are shared periodically with federated averaging, and agents pool in a global policy without revealing local data. This enhances decentralized learning efficiency, speeds up policy convergence, and improves coordination in intricate vehicular routing scenarios. Federated Averaging The experimental framework for evaluating the Multi-Agent Federated Deep Q-Learning (MAQDRL) system is meticulously designed to simulate a realistic and challenging vehicular routing environment. The setup comprises multiple autonomous agents operating within a discretized two-dimensional grid space, where each agent independently learns navigation strategies while collectively contributing to a federated model. Agents are initialized with unique starting positions and assigned individual goals, emphasizing decentralized decision-making and collaboration. To ensure computational efficiency and scalability, a dynamic QuadTree structure is integrated, enabling precise collision detection and effective spatial partitioning. The evaluation examines key metrics, including collision avoidance, energy efficiency, and policy convergence, under a variety of hyperparameter settings and environmental configurations. By implementing federated learning, the framework ensures privacy-preserving parameter aggregation across agents, fostering shared learning while retaining decentralized autonomy. The system verifies its performance against state-of-the-art multi-agent rl algorithms which validates MAQDRL as an effective solution for complex real-world coordination obstacles. The decentralization process occurs through periodic updates of local Q-network parameters \(\phi _i\) through federated averaging protocols.The \(\phi _i\) is denoted in (7) The federated averaging operation computes global parameters at synchronization intervals \(t_k\) which are combined with \(N\) total agents. The aggregated learning outcome from all agents operates without data disclosure thereby ensuring system expansion while maintaining privacy. The federated server obtains parameters from each agent and conducts the averaging calculation as shown in Fig. 1. Global Policy Update Every agent updates its neighborhood Q-network through the following calculation in (8) Agents update their local models through reception of global parameters that generate collective learning benefits which promote both performance and faster convergence speed. Figure 1 shows the agents’ regular connection to the central aggregator (federated server) to get the new global model. Insights of FL and QuadTree Integration QuadTree spatial indexing provides both theoretical and practical advantages when integrated with federated Deep Q-Learning in MAQDRL. First, under the standard contraction-mapping assumptions for DQN [8, 9], each agent’s local Q-update remains a contraction in expectation. Periodic federated averaging, applied to the gradient-descent update defined in (28) and following the formulation in [7], preserves this contraction property by reducing the variance of individual updates through population-level aggregation. As a result, the global model maintains convergence behaviour similar to that of a single-agent DQN trained on stable data, which aligns with the stable empirical learning curves observed in MAQDRL. Beyond variance reduction, the QuadTree structure helps counteract multi-agent non-stationarity by limiting each agent’s interactions to a small, spatially relevant neighbourhood. Because collision checks depend on a logarithmic-size local region rather than the full agent population, the effective Lipschitz constant of the Bellman operator grows much more slowly with \(N\). This leads to smoother temporal-difference (TD) targets and reduces fluctuations in Q-updates, offering an intuitive theoretical basis for the improved training stability observed in dense multi-agent settings. Second, by inserting agent positions into a dynamic QuadTree before collision checks, the framework replaces naïve pairwise comparisons \(\mathcal {O}(N^2)\) with efficient spatial queries of complexity \(\mathcal {O}(\log N)\). This hierarchical spatial partitioning substantially reduces the per-step computational cost and allows the system to scale to much larger groups of agents without introducing additional update noise or communication load. Together, these properties ensure that the combination of federated learning and QuadTree indexing delivers both sound convergence behaviour and practical scalability, enabling MAQDRL to maintain stable learning dynamics even in high-density environments. Assumptions and Constraints The model assumes that agents start at random positions within the environment \(\mathcal {G}\), each with a unique goal, and that no static obstacles are present. These constraints simplify the environment by removing complexities such as overlapping goals and obstacle navigation. This allows the focus to remain on dynamic interactions between agents, such as collision avoidance and efficient pathfinding. Figure 1 illustrates these assumptions, showing agents in an open grid environment with clear paths to their goals. The Proposed Multi-Agent Federated Deep Q-Reinforcement Learning (MAQDRL) Methodolgy The introduced framework, termed Multi-Agent Federated Deep Q-Learning (MAQDRL), addresses the challenges of autonomous multi-agent navigation by integrating reinforcement learning with federated learning and spatial optimization. Each agent independently trains a local Deep Q-Network (DQN) to make navigation decisions based on its state, including position, goal distance, and proximity to other agents. The reward function incentivizes efficient goal attainment and penalizes collisions, encouraging safe and optimal behavior.Federated learning is used to enhance coordination among agents. Periodically, agents share their model parameters, which are aggregated into a global model via federated averaging and redistributed to ensure policy alignment while preserving privacy. This decentralized learning approach reduces communication overhead and promotes cooperation in dynamic environments.The framework leverages a QuadTree-based spatial partitioning technique for efficient collision detection and proximity queries. By dynamically subdividing the grid based on agent density, computational resources are focused on high-interaction areas, enabling scalability for larger agent populations.Additional enhancements include state normalization for stable training and reward scaling for consistent learning dynamics. This methodology provides a scalable and efficient solution for autonomous multi-agent systems, ensuring collision-free navigation and energy optimization. Data Acquisition and Preprocessing The dataset used for this research is derived from CVRPTW-style routing instances [31]; however, the present MAQDRL environment does not solve the complete Capacitated Vehicle Routing Problem with Time Windows (CVRPTW). Specifically, depot assignment, vehicle-capacity constraints, customer demand, service duration, route-load feasibility, and hard time-window constraints are not explicitly modeled in the current experimental setup. Instead, the coordinate information is used to construct a normalized two-dimensional spatial navigation benchmark for multi-agent collision-aware routing. The dataset includes the positions of each agent in the 2D Euclidean space, as well as the distance to their goal. To prepare the data for training, we normalize the agent positions to the range \([0, 100]\), making them consistent with the grid used in our simulation. Each agent’s goal location is also normalized in the same manner, ensuring that the environment’s coordinates are appropriately mapped for the Q-learning process. Therefore, the reported success rate, collision frequency, and energy-consumption results should be interpreted as performance outcomes for CVRPTW-derived multi-agent spatial navigation rather than as complete CVRPTW route-optimization or feasibility results. This clarification ensures that the scope of the proposed framework is limited to decentralized point-to-point navigation, collision avoidance, movement efficiency, and federated policy learning. Non-IID Data Distribution Since MAQDRL follows a federated reinforcement learning setting, data heterogeneity arises from agent-specific interaction histories rather than from supervised class-label imbalance. Each agent interacts with the environment independently and stores its own local transitions in an individual replay memory. Therefore, the local training data of different agents are naturally non-IID because agents start from different positions, move toward different goal locations, encounter different neighbouring agents, and experience different collision and reward patterns. The local replay memory of agent i is represented as: where \(s_i^t\) is the local state, \(a_i^t\) is the selected action, \(r_i^t\) is the received reward, and \(s_i^{t+1}\) is the next state of agent i at timestep t. The local data distribution of agent i can be expressed as: Because each agent observes different trajectories, local neighbourhoods, collision events, and reward sequences, the local distributions are generally heterogeneous: In the experimental setting, this non-IID behaviour is produced through the following factors: - Different initial positions: agents are initialized at distinct random locations in the grid. - Different goal locations: each agent is assigned an individual target location. - Different local neighbourhoods: agents interact with different nearby agents depending on their spatial positions. - Different collision histories: each agent experiences different collision-risk situations during training. - Different reward distributions: rewards vary according to goal progress, collision avoidance, movement cost, and local congestion. During training, no raw trajectories, observations, or replay-buffer samples are exchanged among agents. Only local model parameters are periodically transmitted to the federated aggregator and updated through FedAvg. Thus, the experimental setup follows a naturally non-IID federated reinforcement learning scenario while preserving local data privacy (Table 3). Preprocessing Methods The dataset readiness depends heavily on preprocessing techniques which help the MAQDRL framework perform better and demonstrates more stable learning. The transformation of raw data through these methods creates an appropriate format for training and solves problems like mismatched magnitudes and discrete control decisions and reward fluctuations. The preprocessing enables the reinforcement learning model to maximize dataset utilization so it delivers better training results and provides enhanced system performance. State Normalization The implementation of state normalization served to normalize all state features across their scales. Application of the transformation described in (12) normalizes the state features \(\textbf{S}\) to \([0, 1]\) which ensures large-scale features do not overshadow the learning of other state variables. The essential normalization technique makes sure each state variable shares equal responsibility in model decisions which helps training convergence. The implementation of normalization leads to more stable gradient calculations throughout the neural network preventing the occurrence of exploding or vanishing gradients. The algorithm transforms \(s_{i,j}^v\) (the \(k\)-th feature value of agent \(i\) at \(v\)) using the range of \(s_{k}\) across all records through \(\max (s_{k})\) and \(\min (s_{k})\) normalization. Action Encoding A one-hot encoding technique was used for converting entity actions \(\mathcal {A}_i\) to binary representation vectors. The performance of the model largely depends on this representation because it safeguards against possible errors in numerical action comparisons. Without encoding the model could wrongly assume the numerical nearness between "up" (\(1\)) and "down" (\(2\)) which would affect its learning process. The binary vector format provides a structure that makes the model recognize each action independently and equally probable.For "up" action given in (13). and similarly for other actions. Reward Scaling Reward scaling introduces a standardization mechanism which stabilizes learning by normalizing all the reward values \(\textbf{R}\). By setting R Based on its statistical values including mean \(\mu _R\) and standard deviation \(\sigma _R\) in (14), the method deals with reward distribution variability. The distribution normalization method protects training updates from distortions that could stem from unusually large or small reward values. The normalization of rewards to a standard range enables agents to better distinguish outcome variation so they optimize their policies better while maintaining consistent convergence. The calculation uses \(\mu _R\) as the mean value and \(\sigma _R\) as the standard deviation measurement for the vector \(\textbf{R}\). The preprocessing methods combined to form a reliable dataset which supports efficient learning processes inside MAQDRL. The methods become crucial for agents to obtain precise environmental perception and action assessment information leading to policy improvements necessary for mastering multi-agent routing tasks. QuadTree Partitioning The QuadTree structure \(\mathcal {T}\) in (15)accepted agent positions \((x_i^t, y_i^t)\) in every timestep so agents could fetch results quickly during collision detection. Temporal Data Augmentation Each episode’s state-action trajectory was augmented (16) with synthetic noise to improve robustness: where \(\eta\) is Gaussian noise with variance \(\sigma ^2 = 0.01\). MDP Formulation In the proposed methodology, the multi-agent navigation challenge is modeled as a Markov Decision Process (MDP), enabling the application of reinforcement learning techniques. An MDP provides a formal framework to represent decision-making problems where outcomes are determined by a combination of randomness and the decisions made by agents.It is defined by a tuple \((\mathcal {S}, \mathcal {A}, \mathcal {P}, \mathcal {R}, \gamma )\), where \(\mathcal {S}\) is the state space, \(\mathcal {A}\) is the action space, \(\mathcal {P}\) represents the state transition probabilities, \(\mathcal {R}\) is the reward function, and \(\gamma\) is the discount factor. State Space \(\mathcal {S}\) The state space \(\mathcal {S}\) represents all possible states that an agent can inhabit. For agent \(a_i\) at time \(t\), the state \(s_i^t \in \mathcal {S}\) is defined as: where \(x_i^t\) and \(y_i^t\) are the agent’s current coordinates in the grid \(\mathcal {G}\), \(d_{goal}^i\) is the Euclidean distance to the agent’s goal location as given by (3), \(t\) represents the current timestep, and \(\text {proximity}_i^t\) denotes the average distance to nearby agents within the collision boundary \(B_i^t\). This state representation provides comprehensive information about the agent’s current status, progress towards its goal, temporal context, and awareness of nearby agents for collision avoidance. Action Space \(\mathcal {A}\) The action space \(\mathcal {A}\) encompasses all possible actions that an agent can perform. For an agent \(a_i\), the action set \(\mathcal {A}_i \subset \mathcal {A}\) is: Each action represents moving the agent one unit in a specific direction or staying stationary. This discrete set of actions enables the agent to navigate the grid towards its goal while making decisions to prevent collisions with other agents.Note that the action space is strictly discrete, defined as \(\mathcal {A}_i = \{\text {up}, \text {down}, \text {left}, \text {right}, \text {stay}\}\).No continuous control is considered in this study, and extending the framework to continuous action domains is part of future work. State Transition Probabilities \(\mathcal {P}\) In (19) state transition probability \(\mathcal {P}\) defines the environment’s dynamics by specifying the likelihood of transitioning to a new state \(s'\) given the current state \(s\) and action \(a\). Since the agent’s movements are deterministic, governed by the state transition function \(f(s, a)\) as defined in (19), these transition probabilities simplify to: This determinism in state transitions reduces the complexity of modeling the environment’s dynamics and simplifies the learning process for the agents. State–Action Dimensionality and Complexity Analysis The state space and action space used in MAQDRL are defined in (17) and (18), respectively. From (17), each agent observes a five-dimensional local state vector consisting of its current position, goal-distance information, timestep information, and local proximity information. Therefore, the local state dimension for each agent is: Similarly, (18) defines a discrete action space with five possible movement decisions: up, down, left, right, and stay. Hence, the action-space cardinality for each agent is: For a system with N agents, the theoretical joint action space grows exponentially as \(5^N\). However, MAQDRL does not enumerate this joint action space. Instead, each agent independently evaluates its own local Q-values through the DQN approximator defined in (28). Thus, each local policy network outputs five Q-values corresponding to the five available actions. The computational overhead of local decision-making is mainly determined by the forward pass of the SAF-QNet policy network, while collision-neighbourhood querying is handled through the QuadTree structure defined in (15). Under balanced spatial partitioning, the QuadTree-based neighbourhood query requires \(O(\log N)\) time. Therefore, the per-timestep computational complexity for all agents can be expressed as: where d denotes the number of trainable parameters in the local SAF-QNet policy network. During federated synchronization, the MEC-side aggregator performs element-wise averaging of local model parameters using the FedAvg operation defined in (7). Therefore, the aggregation complexity per synchronization round is: This analysis clarifies that MAQDRL avoids exponential joint-action enumeration and maintains lightweight local inference with linear federated aggregation overhead. Reward Function \(\mathcal {R}\) The reward function \(\mathcal {R}: \mathcal {S} \times \mathcal {A} \times \mathcal {S} \mapsto \mathbb {R}\) offers feedback to the agent based on the action executed and the resulting state. For agent \(a_i\), the reward \(R_i^t\) at time \(t\) is defined as: where \(R_{\text {goal}}\) is a positive reward for reaching the goal, \(\delta _{\text {goal}}^i\) is an indicator function that takes the value 1 if the agent successfully reaches its goal and 0 otherwise, \(R_{\text {collision}}\) is a penalty for collisions, \(\delta _{\text {collision}}^i\) is an indicator function that equals 1 if a collision has occurred and 0 otherwise, and \(f(d_{goal}^i)\) is a proximity-based reward encouraging the agent to move closer to its goal. Equation 24 is designed to influence the agent’s behavior by balancing goal achievement, collision avoidance, and efficient navigation. Policy \(\pi\) Through the policy \(\pi : \mathcal {S} \rightarrow \mathcal {A}\), agents choose particular actions for each detected state. Each agent inside the MAQDRL framework makes use of a Deep Q-Network (DQN) as an estimator for the best policy \(\pi ^*\). Equation 25 represents the optimal action selection based on the action-value function. Objective Function Finding the optimal policy \(\pi ^*\) corresponds to maximizing an agent’s expected cumulative discounted return: where \(\gamma \in [0,1]\) is the discount factor balancing immediate versus future rewards in ( 26) Bellman Equation The Bellman Optimality Equation defines the optimal action-value function \(Q^*(s, a)\) as: where \(s'\) is the next state resulting from taking action \(a\) in state \(s\). Equation 27 provides a fundamental principle for dynamic programming approaches. Deep Q-Network (DQN) Approximation Given the continuous and high-dimensional characteristics of the state space \(\mathcal {S}\), we employ a function approximator \(Q(s, a; \phi )\) parameterized by neural network weights \(\phi\) to estimate the action-value function. The network parameters are adjusted by minimizing the loss function \(J(\phi )\): where the target \(y_i\) is defined as: Equation 28 minimizes the prediction error using the target update in (29). Although Double DQN can reduce overestimation bias by decoupling action selection from target-value evaluation, standard DQN is adopted in the proposed MAQDRL framework due to the structure and deployment requirements of the present routing scenario. The action space is small and discrete, consisting of five movement actions: up, down, left, right, and stay. In addition, the learning process already incorporates experience replay, target-network updates, state normalization, reward scaling, and \(\epsilon\)-greedy exploration to improve training stability. Since the main objective of MAQDRL is to integrate federated learning with QuadTree-based spatial indexing for lightweight collision-aware routing, standard DQN provides a suitable trade-off between learning stability, computational simplicity, communication efficiency, and edge-computing feasibility. Double DQN can be incorporated as a future extension for larger or more complex action-space environments. Experience Replay Experience replay buffers retain transitions \((s, a, R, s')\), enabling agents to learn from prior experiences sampled uniformly at random. This method disrupts the temporal correlations between successive samples, increasing data efficiency and lowering the variance of updates. Equation 28 benefits from experience replay by stabilizing training through diverse learning samples. Federated Learning in MDP Context In the federated learning setup, each agent independently learns its local policy \(\pi _i\) while periodically sharing model parameters to compute a global model \(\phi\): Equation 7 aggregates model parameters to align agents’ policies while preserving data privacy. Neural Network Architecture for Agent Training Inside the MAQDRL framework each agent deploys SAF-QNet to operate on \(Q(s, a; \phi )\) as the optimal action-value function according to [8]. The subsequent part explains the architecture together with operational aspects of SAF-QNet. SAF-QNet,as shown in Fig. 2 functions as a crucial component to give agents effective navigation capabilities in complex settings while enabling them to learn optimal policies through collaboration. The design objective of SAF-QNet integrates high-dimensional state inputs to identify spatial as well as temporal features that guide decision-making. Within MAQDRL this architecture enables agents to develop approximations of \(Q(s, a; \phi )\) which represents the expected reward value for specific actions in present states. SAF-QNet operates through three essential elements which include a receptive input section joined by two activating hidden cells and a result output cell. The system contains multiple layers that efficiently handle state information about agent location as well as team member proximity and goal distance and current time position. During the input layer procedure the system converts state data points into vectorized arrays while the hidden layers perform feature extraction operations. The network’s intermediate layers use structural design to detect fundamental spatial patterns together with essential temporal patterns that agents need when operating in environments with multiple agents. Q-values from the output layer allow the agent to identify the most favorable action from its available choices according to highest Q-value. The developed policy pursues maximum cumulative long-term rewards during constrained interactions between agents through vectorized inputs. The SAF-QNet system applies experience replay together with target networks to stabilize training processes toward proper convergence. Through federated synchronization each agent contributes to a collective model which promote collective learning without jeopardizing information privacy. The computational speed of the SAF-QNet represents one of its main favorable aspects. Lightweight elements in the architectural design guarantee easy scalability for usage across multiple agent networks. Its spatial features which use proximity data work perfectly with environmental QuadTree spatial partitioning to boost performance in dynamic multi-agent operations. The network architecture handles specifically the difficulties which emerge from multi-agent reinforcement learning in densely populated dynamic environments. Fully connected layers employed with federated learning capabilities let the model both observe relationships between spatial and temporal events while keeping its scaling potential and protecting data ownership. The QuadTree structure integration along with the model provides efficient spatial query capabilities which make it highly appropriate for applications like collision detection and path optimization. The system design strikes a performance-solid choice between processing efficiency and stable decision outputs which supports operational flexibility in multiple urban circumstances. The neural network starts with an input layer that processes the state vector \(s_i^t\) for agent \(a_i\) at time t (30): where \((x_i^t, y_i^t)\) represents the agent’s current position, and \(d_{goal}^i\) is the Euclidean distance to its target, defined as (3): Here, t indicates the current timestep, and \(\text {proximity}_i^t\) refers to the average distance to nearby agents detected using the QuadTree structure. The hidden layers capture spatial and temporal features essential for decision-making. The first hidden layer transforms the input state into feature representations (32): where \(W_1 \in \mathbb {R}^{5 \times 64}\), \(b_1 \in \mathbb {R}^{64}\), and \(\sigma (x) = \max (0, x)\) is the ReLU activation function. Here, the superscript in \(\mathbb {R}^{64}\) denotes that \(h_1\) is a 64-dimensional vector. Following this, the second hidden layer continues to abstract these features (33): where \(W_2 \in \mathbb {R}^{64 \times 32}\) and \(b_2 \in \mathbb {R}^{32}\). Similarly, the superscripts in \(\mathbb {R}^{64 \times 32}\) indicate that \(W_2\) is a matrix with 64 rows and 32 columns. The output layer computes the Q-values for each action (34): where \(W_3 \in \mathbb {R}^{32 \times 5}\) and \(b_3 \in \mathbb {R}^5\). The model is trained by minimizing the temporal difference (TD) error (35): where \(\phi ^-\) represents the target network parameters [8]. The training process incorporates multiple mechanisms to enhance learning efficiency and stability. An experience replay buffer [8] stores transition tuples \((s, a, R, s')\), enabling the model to learn from a diverse set of past experiences. To stabilize training, a target Q-network \(Q(s, a; \phi ^-)\) is utilized, with its parameters periodically updated. Federated synchronization is employed to aggregate local parameters \(\phi _i\) from all agents after every K episodes, leveraging the federated averaging approach (36): where N represents the total number of agents. The computational cost of a forward pass is calculated as follows. For the first hidden layer, the parameter count is \(5 \times 64 + 64 = 384\). In the second hidden layer, the parameters total \(64 \times 32 + 32 = 2080\). The final output layer includes \(32 \times 5 + 5 = 165\) parameters. Consequently, the total number of parameters per agent is \(|W| + |b| = 384 + 2080 + 165 = 2629\). For N agents, the overall computational complexity for a forward pass is \(O(N \cdot 2629)\). Convergence and Complexity While rigorous proofs of convergence for Q-learning with nonlinear function approximation remain an active area of research, our MAQDRL algorithm inherits convergence-in-expectation under federated averaging from McMahan et al.[7], provided the local loss functions are smooth and their gradients are bounded. Moreover, each QuadTree collision-query executes in \(O(\log N)\), each DQN back-propagation in \(O(d)\), and the global federated averaging step in \(O(Nd)\) every \(K\) episodes. Because QuadTree indexing restricts each agent’s transitions to a small local neighbourhood, the non-stationarity induced by other agents is significantly reduced. This results in smoother TD-targets and a tighter effective contraction factor for the Bellman operator, providing an informal justification for the improved stability and faster empirical convergence of MAQDRL.Thus, the overall per-timestep complexity is \(O\bigl (N(d + \log N)\bigr )\), which empirically remains tractable for \(N\). Algorithmic Workflow and Optimization Objective The algorithm 1 implements a multi-agent federated Deep Q-Learning procedure whose objective is to maximize the cumulative discounted reward expected for each agent subject to collision avoidance and boundary restrictions. The algorithm proceeds in four phases. Initialization At \(e=0\), each of the \(N\) agents is assigned random start coordinates \((x_i^0,y_i^0)\) and goal \((x_\textrm{goal}^i,y_\textrm{goal}^i)\). Local Q-network parameters \(\phi _i\) and target networks \(\phi _i^-\) are initialized randomly, each replay buffer \(\mathcal {D}_i\) is emptied, and the exploration rate is set to \(\epsilon _0\). A QuadTree \(\mathcal {T}\) covering the full grid (capacity \(C\)) is constructed to enable \(O(\log N)\) collision queries. Episode Loop For each episode \(e=1,\dots ,E\), the environment and QuadTree are reset, agents are placed at their initial positions, and \(\mathcal {T}\) is populated with these locations. Interaction & Learning Over \(T\) timesteps per episode, each agent \(i\) (1) selects action \(a_i^t\) via an \(\epsilon\)-greedy policy on \(Q(s_i^t,a;\phi _i)\), (2) updates its position and reinserts it into \(\mathcal {T}\) to perform a local neighbourhood query for collisions, (3) receives reward \(R_i^t\) (penalizing collisions or rewarding goal-proximal moves), and (4) stores \((s_i^t,a_i^t,R_i^t,s_i^{t+1})\) in \(\mathcal {D}_i\). After the interaction, each agent samples a minibatch from \(\mathcal {D}_i\) and computes the temporal-difference (TD) target as (37). We then update \(\phi _i\) by descending the gradient of the loss \(\tfrac{1}{B}\sum (y_i - Q(s_i^t,a_i^t;\phi _i))^2\), which corresponds to the standard DQN update defined in (28). Federated Aggregation & Decay Every \(K\) episodes, the central server averages local weights, \(\phi \leftarrow \tfrac{1}{N} \sum _{i=1}^N \phi _i\), and redistributes them, \(\phi _i \leftarrow \phi\) for all \(i\), thereby propagating each agent’s local policy refinements (e.g., improved collision avoidance) across the fleet while preserving data privacy. Finally, the exploration rate is annealed, \(\epsilon \leftarrow \max (\epsilon _{\min }, \epsilon \cdot \epsilon _d)\), to shift gradually from exploration to exploitation. At the conclusion of training, the set \(\{\phi _i\}\) constitutes each agent’s optimized policy under the objective (26). Algorithm Key points and Challenges Addressed The MAQDRL algorithm addresses several critical challenges in multi-agent vehicular routing, ensuring decentralized autonomy, energy efficiency, scalability, and collision avoidance. Key challenges and solutions are summarized below: 1. Collision Avoidance In high-density environments, preventing collisions is crucial. The algorithm uses a QuadTree-based spatial partitioning, reducing collision detection complexity to \(O(\log N)\) per query [5]. Each agent maintains a collision boundary: Collisions are penalized in the reward function: where \(\delta _{\text {collision}}^i = 1\) if \(B_i^t \cap B_j^t \ne \emptyset\) for any \(j \ne i\). 2. Decentralized Decision-Making Agents independently select actions using an \(\epsilon\)-greedy policy [9]: Federated parameter synchronization ensures global policy alignment in 7 3. Scalability The algorithm scales efficiently with \(N\) agents. QuadTree queries for collision detection operate in \(O(\log N)\) complexity [5]. Parameter aggregation follows \(O\left( \frac{T}{K} \cdot N \cdot d\right)\), where \(d\) is the Q-network parameter count. Replay buffer sampling has a complexity of \(O(S \cdot d)\), with \(S\) as the minibatch size. 4. Energy-Efficient Routing The reward function incentivizes shorter paths and penalizes unnecessary movements: 5. Real-Time Adaptability The algorithm dynamically updates states and QuadTree partitions: An adaptive \(\epsilon\)-greedy exploration schedule ensures policies refine over time: 6. Policy Alignment Across Agents Federated synchronization aligns Q-networks across agents, reducing variance and ensuring efficient learning [7]: Experimental Setup The environment for evaluating MAQDRL is modeled as a two-dimensional Euclidean space \(\mathcal {G} \subset \mathbb {R}^2\), discretized into a \(100 \times 100\) grid. The spatial bounds are defined as \(\mathcal {G} = \{(x, y) \mid x, y \in [0, 100]\}\). The system includes \(N = 10\) agents, denoted \(\mathcal {A} = \{a_1, a_2, \dots , a_N\}\), which are randomly initialized within \(\mathcal {G}\). Each agent \(a_i\) is assigned a unique goal \((x_{goal}^i, y_{goal}^i)\), sampled uniformly from \(\mathcal {G}\) while ensuring no overlap with other agents’ initial positions. Each agent maintains a collision boundary \(B_i^t\), with collisions detected as per Equation (6). At every timestep, the QuadTree \(\mathcal {T}\) dynamically updates to reflect agent movements, as specified in Equation (15). Agents are initialized with random positions \((x_i^0, y_i^0)\) and unique goals \((x_{goal}^i, y_{goal}^i)\), starting in a collision-free state with their energy counters set to zero. During each simulation step, the environment tracks collisions and updates agent states accordingly. The experimental setting follows the naturally non-IID federated reinforcement learning configuration described in Section 3.6.2, where each agent maintains an independent replay buffer generated from its own start–goal trajectory, local neighbourhood interactions, collision events, and reward sequence. This study utilized a combination of advanced hardware and specialized software to support the computational demands of multi-agent federated reinforcement learning. The hardware setup included an Intel Xeon E5-2690 CPU running at 2.60 GHz, an NVIDIA Tesla V100 GPU with 32GB of memory, and 128GB of RAM. These resources provided the necessary computational power to manage the complexity of the training process and data-intensive tasks associated with the framework. On the software side, Python 3.9 was the primary programming language used to implement the simulations and algorithms. Several libraries were integrated to enhance functionality: NumPy was employed for numerical computations such as state normalization and matrix operations, and Matplotlib was used to create visualizations for training progress, reward trends, and evaluation metrics. PyTorch was leveraged for the training and inference of the SAF-QNet architecture, facilitating tasks like replay buffer sampling and gradient updates [13]. PySyft handled federated learning operations by managing parameter synchronization and aggregation [14], while SciPy was used for statistical analyses, including confidence interval calculations and paired t-tests. System Hyperparameters The system hyperparameters are meticulously selected to optimize the performance and convergence of the Multi-Agent Federated Deep Q-Learning (MAQDRL) framework within the simulated vehicular routing environment. The learning rate (\(\alpha\)) is equal to 0.001, governing the magnitude of updates during the neural network training process, thereby ensuring stable and gradual convergence. The discount factor (\(\gamma\)) is established at 0.95, balancing the significance of immediate rewards against future gains, which is crucial for agents to develop strategies that account for long-term objectives. To manage the balance between exploration and exploitation, the initial exploration rate (\(\epsilon _0\)) is initialized at 1.0, promoting extensive exploration in the early training phases. This rate decays exponentially with a factor of \(\epsilon _d = 0.995\) per episode, reaching a minimum exploration rate (\(\epsilon _{\min }\)) of 0.1, thus encouraging exploitation of learned policies as training progresses. The QuadTree capacity (C) is configured to 4, determining the maximum number of agents within a single QuadTree node before subdivision occurs, which enhances the efficiency of collision detection mechanisms. The synchronization interval (K) is set to 5 episodes, dictating the frequency at which federated model parameters are aggregated across agents, facilitating collaborative learning while maintaining decentralized autonomy. The training regimen comprises 100 episodes (E), each limited to a maximum of 100 steps (T), balancing the need for comprehensive policy learning with computational resource constraints. Additionally, the environment accommodates 10 agents (N), providing a sufficiently complex scenario to evaluate multi-agent interactions without imposing excessive computational demands. The grid dimensions (\(\mathcal {G}\)) are defined as \(100 \times 100\), offering ample spatial coverage for agent navigation and interaction. These hyperparameter settings are summarized in Table 4 and are pivotal in shaping the learning dynamics, ensuring that the MAQDRL framework effectively learns optimal navigation policies while maintaining energy efficiency and collision avoidance. The hyperparameters used in this study were carefully selected based on their relevance to the multi-agent reinforcement learning framework. Each parameter is detailed below, along with the rationale for its selection and its role in the learning process. Learning Rate (\(\alpha\)) The learning coefficient, defined as \(\alpha = 0.001\), determines the update size for the parameters during the neural network’s training cycle. A small value was chosen to ensure stable convergence and prevent overshooting the optimal solution. Mathematically, it determines the magnitude of the updates to the model parameters \(\phi\) during backpropagation: where \(J(\phi )\) represents the loss function. The selected value balances the trade-off between convergence speed and the risk of instability (44). Discount Factor (\(\gamma\)) The discount factor, \(\gamma = 0.95\), controls the weight assigned to future rewards in the cumulative reward computation: A value near ensures that the agent gives significant priority to long-term rewards while still accounting for immediate ones ((45)). This parameter plays a vital role in scenarios requiring sequential decision-making, where future consequences impact present decisions. Initial Exploration Rate (\(\epsilon _0\)) and Decay Rate (\(\epsilon _d\)) The initial exploration rate, \(\epsilon _0 = 1.0\), encourages random actions during the early stages of training to ensure sufficient exploration of the state-action space. Over time, the exploration rate decays exponentially with a decay factor \(\epsilon _d = 0.995\) to promote exploitation of learned policies: where \(\epsilon _{\min } = 0.1\) ensures that some level of exploration is maintained throughout training ((46)). This balance between exploration and exploitation is crucial for discovering optimal policies. Reward Function The reward function integrates goal achievement, collision penalties, and progress toward the goal: Here, \(R_{\text {goal}}\) and \(R_{\text {collision}}\) are positive constants, \(\delta _{\text {goal}}^i\) and \(\delta _{\text {collision}}^i\) are indicator functions, and \(d_{goal}^i\) is the Euclidean distance to the goal ((47)). This configuration of hyperparameters is designed to ensure that the MAQDRL framework efficiently learns optimal policies for collision-free and energy-efficient navigation within a decentralized multi-agent system. Evaluation Metrics The MAQDRL framework evaluation depends on multiple performance indicators to assess agent operations in a multi-agent domain. Collision Rate The MAQDRL framework requires the collision rate measurement to assess its safety and coordination strength in operating in environments where multiple agents interact. The collision rate calculation uses (48) to count all the collisions that agents faced in all time steps. The binary indicator function collisionsi evaluates when an agent i undergoes a collision during time t and assigns a value of 1 to indicate an actual collision event as defined through (6). The framework aims to let agents achieve their tasks through safe movements thus this metric directly represents this core objective. The agents demonstrate effective coordination of their motions when their collision frequency remains low because they learn to avoid mutual interference. The detection system proves that the embedded collision avoidance features in the algorithm function correctly. Assessing the reward structure and the learning process is possible through monitoring this metric for its ability to measure safe navigation behaviors among agents. Energy Consumption Energy consumption is a key metric that evaluates the efficiency of the agents’ navigation strategies within the environment. It is calculated using (49), which sums the Euclidean distances traveled by the agents across all timesteps: where \(\Vert \Delta a_i^t\Vert _2\) denotes the Euclidean norm (or magnitude) of the displacement vector for agent \(i\) at timestep \(t\). This metric aligns with the framework’s objective of minimizing energy usage by incentivizing agents to identify the most efficient routes to their goals. By monitoring energy consumption, we can evaluate the agents’ ability to optimize their movements and avoid unnecessary displacements, which is particularly critical for scenarios with limited or costly energy resources. A reduction in this metric over time signifies that the agents are progressively learning to navigate more efficiently, thereby improving the overall system’s performance and sustainability. Convergence Rate The convergence rate evaluates how quickly the agents’ learning stabilizes and their policies approach optimal solutions. It is expressed in (50) as the change in average reward over a specified number of episodes: where \(\Delta \text {Reward}\) denotes the change in average reward achieved by the agents across \(\Delta E\) episodes. This metric is crucial for analyzing the learning efficiency of the MAQDRL framework in the context of the project. A higher convergence rate indicates that the agents are effectively adapting their policies based on the rewards received, resulting in faster learning and improved performance. Tracking the convergence rate is valuable for assessing the efficiency of the federated learning approach and understanding the impact of factors such as the learning rate and exploration strategy. It also sheds light on whether the agents are leveraging shared knowledge through parameter aggregation to enhance their individual learning processes. A favorable convergence rate demonstrates that the MAQDRL framework is successfully enabling rapid and stable learning in a cooperative multi-agent environment. Comparative Baselines To assess the effectiveness of MAQDRL, its performance is compared against several established multi-agent learning algorithms under identical environmental settings, ensuring a fair and consistent evaluation. The baseline algorithms considered are as follows: Multi-Agent Deep Deterministic Policy Gradient (MADDPG) [10] A framework that employs centralized training with decentralized execution, capable of handling continuous action spaces using deterministic policies optimized through policy gradient methods. Multi-Agent Proximal Policy Optimization (MAPPO) [11] A variation of Proximal Policy Optimization tailored for multi-agent scenarios, emphasizing robustness and stability in cooperative and competitive environments. Multi-Agent Actor-Critic (MARC) [12] A multi-agent adaptation of the actor-critic framework designed to enable agents to learn coordination strategies, either through shared or independent critics. Random Policy A baseline approach where agents choose actions randomly, serving as a reference point to highlight the improvements achieved through learned policies. To ensure a fair comparison, all baselines (MARC, MAPPO, MADDPG) were adapted to operate on the same discrete 5-action grid movement space used by MAQDRL. Each algorithm represents actions using a shared 5-dimensional one-hot encoding corresponding to {up, down, left, right, stay}. For MARC and MAPPO, the policy head outputs a categorical distribution over these five actions, while in MADDPG the original continuous actor output is replaced with a discrete softmax head. Furthermore, all methods use identical network architectures (two-layer MLP with 128–128 hidden units), identical parameter sizes, the same replay buffer capacity, and the same training horizon and optimizer settings. These controls ensure that differences in performance reflect algorithmic behavior rather than architectural or hyperparameter discrepancies. Baseline Implementations To ensure a fair comparison, we implemented three state-of-the-art multi-agent RL algorithms under identical environment, preprocessing, and evaluation settings. Table 5 summarizes the key network architectures, hyperparameters, and training schedules for each baseline. Edge-Computing Relevance: Communication and MEC Load Analysis The edge-computing relevance of MAQDRL is evaluated analytically in terms of model size, communication overhead, and MEC-side aggregation complexity. The local policy network deployed at each agent contains 2629 trainable parameters. Assuming 32-bit floating-point representation, the size of one local model is: In each federated synchronization round, an agent uploads its locally trained model to the MEC-side aggregator and receives the aggregated global model in return. Therefore, the total upload–download communication cost for one agent in a synchronization round is: For N participating agents, the total communication overhead per synchronization round is: In the experimental setup with \(N=10\) agents, the total communication overhead per synchronization round is: If ten synchronization rounds are performed during training, the cumulative communication overhead is approximately: This communication requirement is modest because MAQDRL exchanges only model parameters rather than raw observations, trajectories, sensor streams, or replay-buffer samples. Consequently, the framework reduces communication overhead while preserving the locality of agent-specific data. The MEC-side computation is also limited. Federated averaging requires element-wise aggregation of the local model parameters: The computational complexity of this aggregation step is: where \(d=2629\) denotes the number of trainable parameters in the local policy network. For \(N=10\) agents, the MEC-side aggregator performs: parameter-level aggregation operations per synchronization round. This indicates that the aggregation workload grows linearly with the number of agents and remains computationally lightweight for the evaluated multi-agent setting. Furthermore, local decision-making at each agent requires only a forward pass through a compact policy network that maps a low-dimensional state vector to five discrete action values. Therefore, from an analytical perspective, the proposed framework imposes limited memory, computation, and communication demands on both the agents and the MEC-side aggregator. Overall, the small model size, low synchronization overhead, and linear aggregation complexity indicate that MAQDRL is compatible with MEC-assisted edge-computing constraints. However, this analysis does not replace direct hardware-level validation. Future work will deploy the trained policy on constrained edge devices, such as single-board computers, to measure inference latency, memory footprint, and energy consumption under real execution conditions. Performance Evaluation This section analyzes the performance of the proposed method Multi-Agent Quad Tree based Deep Reinforcement Learning (MAQDRL) model compared to existing models: MARC, MAPPO, MADDPG, and Random Policy. The evaluation is organized in two parts. First, MAQDRL is compared with centralized and non-federated MARL baselines, as summarized in Table 6, to assess its overall performance against established methods. Second, the federated-learning contribution is examined through a separate component-level ablation and federated baseline study in Section 5.4, where Federated DQN is compared with full MAQDRL. The evaluation is based on several key metrics, including cumulative rewards, energy consumption, and collision frequency, with experiments conducted at various learning rates (0.1, 0.01, 0.001) to observe the impact on model performance. The experiments were carried out over 100 episodes, each at 100 maximum steps. A batch size of 32 was used, and epsilon decay rates(\(\epsilon\)) were set at 0.8, 0.9, and 0.995. Each agent’s initial position was randomized, requiring navigation in a multi-agent environment with spatial constraints. A QuadTree structure was employed for efficient collision detection and spatial querying. This structural optimization significantly reduced computational overhead, making the MAQDRL model highly scalable for larger environments with an increased number of agents and achieved a success rate of 97.8%, demonstrating its superior capability in handling complex multi-agent tasks. Table 6 highlights the performance metrics for each model assessed in this study. As shown in Table 6, the MAQDRL model not only surpasses other models in terms of success rate but also excels in managing energy consumption and minimizing collisions, which are crucial for the scalability and efficiency of real-world applications. Reward Analysis of Models Figure 3 illustrates the reward trajectories of various reinforcement learning models across 100 episodes. The models analyzed include the Multi-Agent Quadtree based Deep Reinforcement Learning (MAQDRL, proposed), Multi-Agent Deep Deterministic Policy Gradient (MADDPG), Multi-Agent Proximal Policy Optimization (MAPPO), MARC, and a baseline Random Policy.The reward function, as defined in 47, integrates goal achievement, collision penalties, and progress toward the goal, providing a balanced metric to evaluate the models’ performance. In the initial phase (0-20 episodes), all models display a modest increase in rewards as they begin to learn and adapt to the environment. The proposed MAQDRL model, however, shows a distinct improvement in reward accumulation over the episodes, surpassing the others by a significant margin. By the midpoint of the evaluation period (up to the 60th episode), the MAQDRL model’s rewards rise steeply, achieving about 60, whereas MADDPG and MAPPO reach levels near 40 and 30, respectively. MARC and the Random Policy show slower progress, indicating less efficient optimization strategies. At the end of 100 episodes,as shown in Fig. 3 the MAQDRL model leads with a total reward of approximately 125, highlighting its effective balance between exploration and exploitation and its ability to adapt dynamically. MADDPG, MAPPO, and MARC follow with 85, 68, and 60, respectively, while the Random Policy accumulates the least rewards, slightly above 20. This performance suggests that the MAQDRL model’s advanced algorithms and decision-making capabilities allow it to significantly outperform other models in accumulating rewards in a multi-agent system, making it a promising solution for complex autonomous operational tasks. Collision Frequency Analysis Collision rates are an essential metric in multi-agent environments, reflecting the model’s ability to ensure safe coordination. As shown in Fig. 4, collision frequencies across various reinforcement learning models over 100 episodes are compared. The proposed MAQDRL model consistently exhibits the lowest frequency of collisions, demonstrating its superior navigation and interaction strategies within a multi-agent environment. Over the entire period, MAQDRL recorded only 21 collisions, significantly lower than MADDPG (62), MAPPO (78), MARC (98), and the baseline Random Policy, which shows higher variability with sporadic peaks. The collision rate metric is calculated using (48), which considers the number of collisions per episode to assess the safety and efficiency of models. The graph indicates occasional spikes in collision frequencies for MADDPG, MAPPO, and MARC, suggesting episodic lapses in coordination or decision-making. These models, such as MADDPG and MAPPO, demonstrate moderate effectiveness in managing collision rates, while MARC’s higher frequency suggests a more aggressive or suboptimal collision avoidance mechanism. This evaluation underscores MAQDRL’s strength in not only maximizing rewards but also ensuring greater operational safety, outperforming other models in reducing conflicts within complex environments. Table 7 shows collision frequencies for different models across 25, 50, 75 and 100 episodes. The MAQDRL model consistently records the lowest collision rates, highlighting its effective collision avoidance capabilities. At each interval, MAQDRL’s collision numbers are significantly lower than those of other models, with a gradual increase from 11 at 25 episodes to 31 by the 100th episode. Conversely, the Random Policy model exhibits the highest collision rates, surging from 70 to 192 by the 100th episode, indicating its inefficacy. MARC and MAPPO also show increasing trends in collisions, but MARC’s rates more than double. This table illustrates the effectiveness of the MAQDRL model’s algorithms in managing risks and ensuring safety in multi-agent environments. Energy Consumption Analysis Figure 5 presents the average energy consumption per episode across various models over 100 episodes. The graph reveals that the MAQDRL model consistently exhibits the lowest energy consumption, maintaining values significantly below those of other models, with an average around 265.98 KJ. This efficiency is quantified using the energy consumption metric described in (49), which incorporates factors such as the agents’ movements, actions, and system interactions. In contrast, other models, including MADDPG, MAPPO, and MARC, show higher and more variable energy consumption patterns. MADDPG and MAPPO fluctuate around 604.20 KJ and 611.59 KJ respectively, indicating moderate energy efficiency. MARC, however, shows the highest variability, with peaks reaching up to approximately 498.87 KJ, suggesting less stable energy management. The Random Policy model displays the highest and most erratic energy usage, peaking frequently and averaging 714.93 KJ, which highlights the inefficiencies associated with non-strategic, random decision-making processes. This energy consumption analysis not only highlights the superior energy efficiency of the MAQDRL model but also indicates areas where other models could improve to enhance their operational sustainability in energy-sensitive environments. Component-Level Ablation and Federated Baseline Study To evaluate the contribution of each core module within the proposed MAQDRL framework and to provide a directly comparable federated baseline, we conducted a component-level ablation and federated baseline study using three configurations: (i) DQN + QuadTree, which uses the spatial partitioning structure for efficient collision detection but does not use federated synchronization; (ii) Federated DQN, which uses local DQN training with periodic FedAvg-based parameter synchronization but does not use the QuadTree spatial-indexing module; and (iii) full MAQDRL, which integrates DQN-based local learning, federated parameter sharing, QuadTree-based spatial indexing, and reward shaping. Figure 6 presents the reward trajectories across the three variants. The Full MAQDRL variant consistently outperforms the others, achieving the highest cumulative reward. This is attributed to the synergy of federated updates accelerating policy generalization and the QuadTree structure mitigating navigation inefficiencies. Federated DQN shows a smooth, steady rise in rewards, driven by cross-agent learning but lacks the local spatial efficiency of QuadTree. Meanwhile, DQN + QuadTree sees faster early gains but saturates earlier, reflecting the absence of federated aggregation to propagate successful strategies across agents. As shown in Fig. 7, collision frequency drops significantly with the introduction of the QuadTree. The full model outperforms both ablated variants, as federated sharing helps agents collectively avoid congested paths. Without federated updates, DQN + QuadTree performs well in reducing early collisions but cannot adapt to long-tail interaction patterns across multiple agents. Federated DQN reduces collisions moderately by aligning agent strategies, but the absence of spatial awareness limits its responsiveness to dynamic proximity conflicts. The FedDQN comparison directly evaluates the federated-learning baseline within the same learning paradigm. Since both FedDQN and MAQDRL use local DQN training with FedAvg-based synchronization, their comparison provides a federated-versus-federated evaluation. The stronger performance of MAQDRL over FedDQN indicates that the improvement is not only due to federated parameter sharing, but also due to the QuadTree-based spatial indexing module, which improves local neighbourhood querying and collision-aware decision-making. Energy usage results in Fig. 8 highlight the efficacy of reward shaping. The Full MAQDRL model learns compact and energy-efficient trajectories, maintaining an average energy level nearly 20–30% lower than its ablated counterparts. In contrast, DQN + QuadTree lacks an explicit incentive to optimize for energy, while Federated DQN, though collaborative, expends more energy navigating without spatial optimization. This confirms that only the full integration of all three elements achieves optimal trade-offs across stability, safety, and sustainability. In summary, the ablation results underscore that each proposed innovation independently contributes to performance gains, while their combination in MAQDRL yields the most robust and generalizable multi-agent routing policy. Analysis of learning rate (\(\alpha\)) on Model Performance Table 8 offers a detailed overview of the proposed model’s performance metrics across different learning rates (\(\alpha\)), showcasing variations in total rewards, total collisions, and average energy consumption. The corresponding graphical representations further elucidate these differences: total rewards are illustrated in Fig. 9, total collisions in Fig. 10, and average energy consumption in Fig. 11. The analysis reveals that the proposed learning rate of 0.001 offers superior performance, achieving the highest total reward of 4736, the lowest collision count of 21, and the most efficient energy consumption at an average of 2601 KJ. These outcomes highlight the learning rate’s ability to optimize the balance between reward maximization and safety-critical metrics like collision minimization and energy efficiency. By contrast, higher learning rates (\(\alpha\)) of 0.1 and 0.01, while delivering reasonably high reward levels of 4275 and 4695, result in increased collision rates (39 and 27, respectively) and slightly higher energy consumption (2720 KJ and 2660 KJ). These findings underscore the effectiveness of the 0.001 learning rate in achieving robust performance suitable for multi-agent environments where operational safety and sustainability are critical. The reward curves demonstrate that with \(\alpha =0.1\), the policy learns quickly but exhibits high variance and occasional declines after episode 60, indicating instability from large gradient steps. With \(\alpha =0.01\), the growth is steadier and plateaus around 50–60 reward units, evidencing more balanced updates. At \(\alpha =0.001\), the trajectory rises smoothly and reaches the highest plateau, confirming that smaller learning steps yield the most consistent long-term improvements. Collision frequencies directly reflect the trade-off between aggression and caution in policy updates. The \(\alpha =0.1\) setting produces frequent spikes (up to four collisions per episode), as agents overcorrect and intersect. Reducing to \(\alpha =0.01\) halves these spikes, and \(\alpha =0.001\) yields near-zero collisions throughout training, showing that conservative updates foster safer multi-agent coordination. Energy consumption trends mirror collision behavior: the high-variance \(\alpha =0.1\) policy incurs energy peaks (often above 300 KJ) due to frequent collision recoveries. The moderate rate (\(\alpha =0.01\)) reduces wasted motion, averaging around 250–300 KJ. The slowest rate (\(\alpha =0.001\)) minimizes unnecessary deviations, stabilizing energy use around 200–250 KJ. This alignment confirms that collision avoidance directly contributes to energy efficiency when learning proceeds with smaller, safer updates. Analysis of Different Epsilon Decay Rates (\(\epsilon\)) Epsilon decay plays a critical role in reinforcement learning by determining the balance between exploration and exploitation as the agent trains. A lower decay rate results in the agent exploiting its learned policy earlier, while a higher decay rate allows for prolonged exploration, potentially leading to better long-term learning outcomes. In this section, we compare the performance of three epsilon decay rates (\(\epsilon _0 = 1\) decayed to \(\epsilon _{\min } = 0.1\) with \(\epsilon _d \in \{0.8, 0.9, 0.995\}\)) across three important metrics: reward accumulation, collision frequency, and energy consumption. These comparisons were performed over a total of 100 episodes. Table 9 shows results of Performance Metrics for Different Epsilon Decay Rates. Figure 12 presents the comparison of reward accumulation across different epsilon decay rates. The graph illustrates that with a decay rate of 0.995, the agent is able to achieve the highest cumulative reward by the end of the training, suggesting that maintaining exploration for a longer period helps the agent to avoid suboptimal strategies and learn better policies. The agents using decay rates of 0.9 and 0.8 show slower convergence and a more erratic reward pattern, indicating that they exploit learned policies too early, missing potential optimizations that could arise from continued exploration. In Fig. 13, we observe the frequency of collisions over the course of the episodes. The data shows that the slower epsilon decay rate of 0.995 leads to fewer collisions, reinforcing the idea that longer exploration helps agents learn safer paths before fully exploiting their strategies. In contrast, the agents with faster epsilon decay rates (0.8 and 0.9) encounter more collisions, likely due to prematurely converging to suboptimal policies that fail to account for all possible obstacles. Figure 14 compares the total energy consumption per episode for the different epsilon decay rates. Although the difference in energy consumption is not drastic, the agents using a slower decay rate of 0.995 consistently consume slightly less energy, averaging around 2588 KJ. This suggests that the extended exploration period helps agents find more efficient paths, minimizing redundant movement and energy expenditure. The faster decay rates show a higher energy consumption, possibly due to the agents engaging in more aggressive exploitation without exploring energy-efficient strategies. Overall, these results indicate that the choice of epsilon decay rate significantly impacts the agent’s ability to balance exploration and exploitation, with a slower decay rate (0.995) proving most effective in achieving better rewards, fewer collisions, and lower energy consumption. The analysis underscores the importance of tuning the exploration schedule for multi-agent reinforcement learning tasks, especially in environments requiring both safety and efficiency. Time Complexity Analysis The time complexity of the proposed MAQDRL model and its comparison with other models, such as MARC, MAPPO, MADDPG, and Random Policy, was analyzed based on their ability to handle spatial queries and decision-making in multi-agent environments. The MAQDRL model demonstrated significant improvements in time efficiency due to the integration of the QuadTree structure for collision detection and spatial querying. The hierarchical partitioning of the environment allowed for logarithmic time complexity for collision checks, making it suitable for large-scale multi-agent systems. Time Complexity Comparison The MAQDRL model achieves a time complexity of \(O(\log N)\) for collision detection and spatial queries due to the QuadTree structure, and \(O(N)\) for policy updates per agent, resulting in an overall complexity of \(O(N \log N)\) for each step. In contrast, MARC has a time complexity of \(O(N^2)\), as it relies on pairwise collision checks for all agents. Similarly, MAPPO exhibits a complexity of \(O(N^2)\) due to centralized policy evaluation and joint action-value computations. The MADDPG model also has a time complexity of \(O(N^2)\), stemming from inter-agent dependency during training and evaluation. Lastly, the Random Policy model operates with a complexity of \(O(N^2)\), as there is no spatial querying or policy computation. Robustness Boundaries and Practical Limitations Although MAQDRL improves collision-aware navigation in the evaluated grid-based setting, its practical performance may vary under more challenging deployment conditions. The main robustness boundaries are summarized in Table 10. These limitations do not contradict the computational scalability of the framework; rather, they identify the conditions under which additional mechanisms may be required. In high-density settings, the QuadTree module reduces the computational cost of neighbourhood querying but cannot remove the physical difficulty of finding collision-free paths in overcrowded regions. Similarly, during communication delays, agents can continue local decision-making using their current DQN policies, but prolonged synchronization gaps may reduce cross-agent policy alignment. Adversarial agents, severe non-IID local experiences, noisy localization, and obstacle-rich environments may also require robust aggregation, asynchronous synchronization, personalized federated learning, and obstacle-aware state representations in future extensions. Conclusion The proposed MAQDRL framework integrates federated learning with QuadTree-based spatial decomposition to support scalable, collision-aware, and energy-efficient multi-agent coordination. Experimental results show that MAQDRL consistently outperforms established MARL baselines such as MARC, MAPPO, and MADDPG in reward maximization, energy efficiency, and collision reduction under varying learning rates and exploration-decay conditions. An optimized learning rate (\(\alpha = 0.001\)) provides stable training while maintaining performance, and a higher exploration decay rate (\(\epsilon = 0.995\)) supports sustained exploration and improved long-term behavior. The present experiments were conducted in a simplified grid environment with ten agents and without static obstacles or multi-goal scenarios. This abstraction was chosen to isolate the core learning dynamics of MAQDRL. The robustness boundaries discussed in Section 5.8 further clarify that additional mechanisms may be required under extreme local density, communication delays, adversarial behaviour, severe non-IID experience distributions, noisy localization, and obstacle-rich environments. These limitations do not negate the computational scalability provided by QuadTree-based spatial indexing or the communication efficiency of federated parameter sharing; rather, they identify the conditions under which the framework should be extended. The current evaluation should also be interpreted as a CVRPTW-derived spatial navigation setting rather than a complete CVRPTW optimization framework, since depot assignment, capacity constraints, route-load tracking, customer demand, service duration, and hard time-window feasibility are not explicitly modeled. Future work will extend MAQDRL toward full CVRPTW-aware routing by incorporating these constraints into the state representation, reward design, and route-validation process. The revised evaluation further includes Federated DQN as a federated MARL baseline through the component-level ablation and federated baseline study. This provides a federated-versus-federated comparison, since both Federated DQN and MAQDRL use FedAvg-based parameter synchronization, while only MAQDRL includes QuadTree-based spatial indexing. More complex federated policy-gradient and actor-critic baselines such as FedPPO and FedMADDPG, along with hybrid optimization-based and machine-learning-based baselines such as ILP and MLP, remain important future extensions. We additionally provide an analytical communication and MEC-load analysis, showing that MAQDRL requires approximately 0.20 MB per synchronization round for ten agents and has linear FedAvg aggregation complexity. These results support the communication efficiency and MEC compatibility of the framework, while direct edge-hardware measurements of inference latency, memory footprint, and energy consumption remain part of future validation. Future work will also investigate obstacle-rich and map-based environments using datasets such as SUMO and CityFlow and scale the agent population beyond fifty to evaluate routing realism and communication efficiency in denser settings. Overall, MAQDRL provides a theoretically motivated and practically scalable foundation for distributed multi-agent reinforcement learning. By combining variance-reducing federated averaging with the locality-aware structure of QuadTree indexing, the framework enables stable, privacy-preserving, and communication-efficient coordination among autonomous agents, paving the way for further validation in larger and more realistic multi-agent environments. Data Availability No datasets were generated or analysed during the current study. References Huang H, Chu X, Liu F. DMITS: Dependency and Mobility-Aware Intelligent Task Scheduling in Socially-Enabled VFC Based on Federated DRL Approach. IEEE Trans Mobile Comput. 2021;21(12):4421–35. https://doi.org/10.1109/TMC.2021.3112716. Zhao N, Ye Z, Pei Y. Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing. IEEE Trans Wireless Commun. 2022;21(9):6949–63. https://doi.org/10.1109/TWC.2022.3153316. Peng B, Wang J, Zhan Z. A Deep reinforcement learning algorithm using dynamic attention model for vehicle routing problems. In: International conference on intelligent computing. Springer: Cham; 2019. p. 160–170. https://doi.org/10.1007/978-3-030-26766-7. Brecko A, Zri I, Klepec J. Federated learning for edge computing: A survey. Applied Sci. 2022;12(18):9124. [Online]. Available: https://doi.org/10.3390/app12189124. Finkel RA, Bentley JL. Quad trees: A data structure for retrieval on composite keys. Acta Informatica. 1974;4(1):1–9. https://doi.org/10.1007/BF00288933. Piorkowski M, Sarafijanovic-Djukic N, Grossglauser M. CRAWDAD epfl/mobility. In: IEEE Dataport. 2022. https://doi.org/10.15783/C7J010. McMahan HB, Moore E, Ramage D, Hampson S, Aguera y Arcas B. Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th international conference on Artificial Intelligence and Statistics (AISTATS). 2017. p. 1273–1282. https://proceedings.mlr.press/v54/mcmahan17a.html. https://doi.org/10.48550/arXiv.1602.05629. Mnih V, et al. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–33. https://doi.org/10.1038/nature14236. Sutton RS, Barto AG. Reinforcement Learning: An Introduction. MIT press. 1998. http://incompleteideas.net/book/the-book.html. https://doi.org/10.5555/551283. Lowe R, Wu Y, Tamar A, Harb J, Abbeel P, Mordatch I. Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in neural information processing systems. 2017. p. 6379–6390. https://doi.org/10.48550/arXiv.1706.02275. Yu C, et al. The surprising effectiveness of PPO in cooperative multi-agent games. 2021. arXiv preprint arXiv:2103.01955 Foerster JN, Assael IA, de Freitas N, Whiteson S. Learning to communicate with deep multi-agent reinforcement learning. In: Advances in neural information processing systems. 2016. p. 2137–2145. https://proceedings.neurips.cc/paper/2016/hash/55b1927fdafef39c48e5b73b5dba77b6-Abstract.html, https://doi.org/10.48550/arXiv.1605.06676. Paszke A, et al. PyTorch: An imperative style, high-performance deep learning library. In: Advances in neural information processing systems. 2019. p. 8024–8035. https://doi.org/10.48550/arXiv.1912.01703. Ryffel T, et al. A generic framework for privacy preserving deep learning. 2018. arXiv preprint arXiv:1811.04017 Everett M, Chen YF, How JP. Collision avoidance in pedestrian-rich environments with deep reinforcement learning. IEEE Access. 2021;9:10357–77. https://doi.org/10.1109/ACCESS.2021.3050338. Ouahouah S, Bagaa M, Prados-Garzon J, Taleb T. Deep-reinforcement-learning-based collision avoidance in UAV environment. IEEE Intern Things J. 2022;9(6):4015–30. https://doi.org/10.1109/JIOT.2021.3118949. Xue Y, Chen W. Multi-Agent Deep Reinforcement Learning for UAVs Navigation in Unknown Complex Environment. IEEE Trans Intell Vehic. 2024;9(1):2290–303. https://doi.org/10.1109/TIV.2023.3298292. Ben Elallid B, Abouaomar A, Benamar N, Kobbane A. Vehicles control: Collision avoidance using federated deep reinforcement learning. In: GLOBECOM 2023 - 2023 IEEE global communications conference. Kuala Lumpur, Malaysia; 2023. p. 4369-4374. https://doi.org/10.1109/GLOBECOM54140.2023.10436831. Kim M, Lee S, Lim J, Choi J, Kang SG. Unexpected Collision Avoidance Driving Strategy Using Deep Reinforcement Learning. IEEE Access. 2020;8:17243–52. https://doi.org/10.1109/ACCESS.2020.2967509. Li X, Chen Y, Li B, Liu W, Wu J. Federated reinforcement learning for decentralized autonomous navigation: A UAV swarm case study. IEEE Intern Things J. 2021;8(20):15129–40. https://doi.org/10.1109/JIOT.2021.3067113. Zhang Q, Wang L, Liu N, Wang M, Chen D. Multi-agent deep reinforcement learning-based collision-aware path planning for UAV swarms. IEEE Trans Intell Transport Syst. early access. 2023. https://doi.org/10.1109/TITS.2023.3256789. Shen J, Sun Y, Li Z. Adaptive multi-agent deep q-learning for dynamic collision avoidance in swarm UAV operations. IEEE Trans Intell Transport Syst. 2023;24(2):2305–17. https://doi.org/10.1109/TITS.2022.3167032. Nguyen N-D, De Silva D. Deep reinforcement learning for multi-UAV routing, coordination, and path planning. IEEE Trans Intell Transport Syst. early access. 2022. https://doi.org/10.1109/TITS.2022.3176390. Liu Z, Yan Z, Cao X, Chen X. Multi-UAV collaborative path planning under uncertain environment using deep reinforcement learning. IEEE Trans Vehic Technol. 2021;70(4):4255–67. https://doi.org/10.1109/TVT.2021.3059415. Wang G, Zhu S, Chen Y, Wang M. Federated multi-agent reinforcement learning for UAV swarm collision avoidance. IEEE Trans Mobile Comput. early access (2023). https://doi.org/10.1109/TMC.2023.3289468. Chen X, Zhang T, Qin Z, Ni Q. Cooperative multi-agent deep reinforcement learning for dynamic UAV navigation. IEEE Intern Things J. 2022;9(15):12877–89. https://doi.org/10.1109/JIOT.2022.3148897. Hou Z, Shi Y, Wang X, Lu C. Towards safe and efficient multi-UAV cooperation: A deep reinforcement learning approach. IEEE Trans Intell Transport Syst. 2020;21(10):4152–62. https://doi.org/10.1109/TITS.2019.2958546. Smith SF, Barlow GJ, Xie X-F, Rubinstein ZB. SURTRAC: Scalable urban traffic control. TRB Ann Meet Paper. 2013. [Online]. Available: https://www.ri.cmu.edu/publications/surtrac-scalable-urban-traffic-control/. Barlow GJ, Smith SF, Xie X-F. Optimizing for everyone using cooperative adaptive control. JCT Traffic Signal Sympos. 2019. [Online]. Available: https://www.jctconsultancy.co.uk/Symposium/Symposium2019/PapersForDownload/Optimizing. Zhang C, Liu X, Yao A, Bai J, Dong C, Pal S, Jiang F. Fed4UL: A cloud–edge–end collaborative federated learning framework for addressing the non-IID data issue in UAV logistics. Drones. 2024;8(7):312. [Online]. Available. https://doi.org/10.3390/drones8070312. Piorkowski M, Sarafijanovic-Djukic N, Grossglauser M. CRAWDAD epfl/mobility. IEEE Dataport. 2022. [Online]. Available: https://doi.org/10.15783/C7J010. Zhou Y, Tan L, Tan W. Federated deep q-network for last-mile route optimization in edge environments: A case study with singpost. In: Proceedings of the 2024 IEEE international conference on edge computing. 2024. [Online]. Available: https://ieeexplore.ieee.org/document/10345678. Huang H, Chu X, Liu F. DMITS: Dependency and mobility-aware intelligent task scheduling in socially-enabled VFC based on federated DRL approach. IEEE Trans Mobile Comput. 2021;21(12):4421–35. https://doi.org/10.1109/TMC.2021.3112716. Zhang K, Yang Z, Başar T. Multi-agent reinforcement learning: A selective overview of theories and algorithms. In: Handbook of reinforcement learning and control. Springer; 2020. p. 321–384. arxiv:1911.10635 Rahim S, Peng L, Ho P-H. TinyFDRL-Enhanced energy-efficient trajectory design for integrated space–air–ground networks. IEEE Intern Things J. 2024;11(12):21391–21401. [Online]. Available: https://doi.org/10.1109/JIOT.2024.3361394. Rahim S, Peng L, Ho P-H. DRL-based energy-efficient trajectory planning for multiple UAVs under centralized control. J Network Netw Appl. 2024;4(3):118–128. [Online]. Available: https://doi.org/10.33969/J-NaNA.2024.040303. Wu L, Lin H, Wang X. Federated Training Generative Adversarial Networks for Heterogeneous Vehicle Scheduling in IoV. In: IEEE Intern Things J. 2025;12(5):4888–4898. https://doi.org/10.1109/JIOT.2024.3506159. Wu L, Zhou T, Lai Y, Lin H, Hu J. Personalized asynchronous federated learning for intelligent vehicular computing. IEEE Trans Consum Electron. 2025;71(2):6271–9. https://doi.org/10.1109/TCE.2025.3534468. Wu L, et al. Toward personalized federated meta-learning with constrained hypernetwork on non-IID data. IEEE Trans Comput. 2026;75(1):233–46. https://doi.org/10.1109/TC.2025.3625233. Funding The work is not funded by any agency or organization either technically or financially. Author information Authors and Affiliations Contributions Author Contribution: All authors contributed equally. Corresponding author Ethics declarations Ethical Approval All authors read before submission, approved the final manuscript for submission, and agreed to the terms and conditions of this journal. Consent for Publication All authors gave explicit consent to publish this manuscript. Competing Interests No competing financial or ethical interests. Additional information Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Rights and permissions Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. About this article Cite this article Raj, A.K., Sharma, A. & Naik, K.J. MAQDRL: QuadTree-Based Multi-Agent Federated Deep Q-Learning for Collision Avoidance and Routing Optimization with MEC-Aware Communication Analysis. Cogn Comput 18, 107 (2026). https://doi.org/10.1007/s12559-026-10647-x Received: Accepted: Published: Version of record: DOI: https://doi.org/10.1007/s12559-026-10647-x