With the increasing prevalence of automated guided vehicles (AGVs) in industry, the AGV dispatching has emerged as a research hotspot. Recently, matrix manufacturing systems (MMS) have been favored for their suitability in mass-customized production. However, fluctuating and diverse transport demands in MMS pose challenges for effective AGV scheduling. This paper investigates a multi-AGV dispatching problem (AGVDP) in MMS, involving time-constrained pick-up and delivery tasks. A mixed-integer linear programming model is developed to minimize total transportation costs, including travel distance, early arrival time, and utilized AGVs. To achieve this, an effective adaptive large neighborhood search algorithm (ALNS) is proposed. In ALNS, a scope-based nearest neighbor heuristic is introduced for initial construction. Diverse problem-specific operators are designed to generate large neighborhoods. To enhance ALNS's robustness to instance variations, an adaptive simulated annealing acceptance criterion is proposed. Computational experiments using a well-established benchmark validate the superiority of the proposed ALNS.