Wearable kinematic sensing enables fine-grained monitoring of hand function, which is important for rehabilitation, assistive technology, and clinical assessment of motor disorders. The ultimate goal is to use it as a digital twin in order to identify specific motor disorders, and monitor their evolution. However, decoding exoskeletal data glove recordings is challenging because the signals are high dimensional, anatomically constrained, and informative mainly through coordinated finger joint dynamics over time. Moreover, the resulting classifiers should be interpretable at the finger joint level. We introduce (i) a \emph{Physio-Digital Temporal Graph (PDTG)} that models hand anatomy and supports correlation derived functional connectivity overlays for interpretation, and (ii) an explainable Temporal Graph Neural Network (TGNN) that performs message passing using Graph Convolutional Network (GCN) at each time step and aggregates temporal dynamics with a bidirectional Long Short Term Memory (bi-LSTM) for window level classification of six hand movement tasks that were performed by 17 younger adults and 17 older adults using an exoskeleton data glove. Raw joint-angle trials are transformed into angle, velocity, and acceleration features, segmented into sliding windows which we evaluate using 5-fold subject-wise cross-validation to prevent subject-leakage. Our TGNN model reached 92.5% ± 2% macro-F1 score for window-level task classification, and for interpretability, we used integrated gradients to produce finger joint level attribution maps and compared attribution patterns between younger and older adults on fine manipulation tasks, highlighting task dependent differences in finger joint involvement. In general, the proposed framework provides an interpretable way to decode hand movements and enables group level comparisons among age groups based on hand topology.