ABSTRACT. Existing state-of-the-art vertical autoscalers for containerized environments are traditionally built for cloud applications, which might behave differently than HPC workloads with their dynamic resource consumption. In these environments, autoscalers may create an inefficient resource allocation. This work analyzes nine representative HPC applications with different memory consumption patterns. Our results identify the limitations and inefficiencies of the Kubernetes Vertical Pod Autoscaler (VPA) for enabling memory elastic execution of HPC applications. We propose, implement, and evaluate ARC-V. This policy leverages both in-flight resource updates of pods in Kubernetes and the knowledge of memory consumption patterns of HPC applications for achieving elastic memory resource provisioning at the node level. Our results show that ARC-V can effectively save memory while eliminating out-of-memory errors compared to the standard Kubernetes VPA.

ABSTRACT. High Performance Computing (HPC) systems rely on fixed user-provided estimates of job time limits. These estimates are often inaccurate, resulting in inefficient resource use and the loss of unsaved work if a job times out shortly before reaching its next checkpoint. This work proposes a novel feedback-driven autonomy loop that dynamically adjusts HPC job time limits based on checkpoint progress reported by applications. Our approach monitors checkpoint intervals and queued jobs, enabling informed decisions to either early cancel a job after its last completed checkpoint or extend the time limit sufficiently to accommodate the next checkpoint. The objective is to minimize tail waste, that is, the computation that occurs between the last checkpoint and the termination of a job, which is not saved and hence wasted. Through experiments conducted on a subset of a production workload trace, we show a 95% reduction of tail waste, which equates to saving approximately 1.3% of the total CPU time that would otherwise be wasted. We propose various policies that combine early cancellation and time limit extension, achieving tail waste reduction while improving scheduling metrics such as weighted average job wait time. This work contributes an autonomy loop for improved scheduling in HPC environments, where system job schedulers and applications collaborate to significantly reduce resource waste and improve scheduling performance.

ABSTRACT. Modern research relies on scientific workflows to manage complex computational tasks across high-performance computing (HPC) systems. Most HPC systems still use a largely static resource allocation strategy, which means resources are fixed during runtime. However, modern workflows are increasingly data and event-driven, especially with the integration of artificial intelligence tasks. Static workflows lack the flexibility to adapt at runtime to evolving workflow and system demands, often impacting resource utilization and overall efficiency.

To address these concerns, scientific workflows require compute elasticity, the capability to dynamically adjust resource usage based on computational load and system availability. Elastic workflows can enhance computational efficiency by leveraging adaptive scheduling algorithms and smart resource management strategies. However, implementing elastic workflows in HPC environments is challenging, as it requires modifications across the entire HPC software stack, including workflow managers, resource managers, and middleware layers. This paper presents the design and implementation of an elastic workflow framework for HPC, highlighting key challenges and design considerations. Our framework is based on an elastic Parsl workflow manager and a custom elastic resource manager, both leveraging capabilities of an implementation of the Process Management Interface for Exascale (PMIx) API as middleware. Through real-world and synthetic workflow case studies, we demonstrate improved resource utilization, reduced execution times, showcasing the benefits of elasticity in HPC workflows.

ABSTRACT. Simultaneous multithreading (SMT) processors improve system throughput by sharing core resources among the threads running on the same core. However, intra-core interference can cause co-running applications to degrade each other's performance significantly.

To address this issue, some approaches have focused on balancing contention at the core shared resources (e.g. the shared L1 data cache). A key advantage of these approaches is that they are workload-agnostic. Other approaches improve the previous ones by modeling the inter-application interference across the intra-core shared resources. Unfortunately, these approaches require off-line model training for specific workloads.

This paper presents WAPA, a CPI-based thread-to-core allocation approach that incorporates the best of both worlds. WAPA is a workload-agnostic policy that implicitly accounts for inter-thread interference across all the shared resources by leveraging the CPI. The proposed approach relies on the optimal transport (OT) theory, a mathematical theory to dynamically select symbiotic pairs of applications.

Experimental results in an Intel Xeon show that WAPA outperforms the default Linux scheduler on average by 8.4% in IPC in workloads dominated by cache and main memory latencies, while performance gains of existing approaches are below 3.2%.

ABSTRACT. Diffusion models have emerged as powerful tools for generative AI tasks. While prior research primarily focuses on eliminating redundancy across timesteps, models like Stable Diffusion introduce a ResNet-Transformer Alternating Execution (RTAE) Pattern, where convolution and attention operators execute sequentially within each timestep. This execution pattern creates two major challenges: (1) excessive on-chip memory access due to frequent data movement between convolution and Transformer operations, and (2) underutilization of computational resources caused by the distinct processing characteristics of these two operators. To tackle these challenges, we propose FDHA, an accelerator designed for efficient diffusion model inference. First, to mitigate redundant on-chip memory access, FDHA introduces an inter-operator dataflow fusion mechanism that strategically aligns ResNet's convolution and Transformer's matrix multiplication dimensions, enabling efficient kernel reuse. Second, to maximize computational resource utilization, FDHA employs a heterogeneous architecture with dedicated Processing Elements for convolutions and Tensor Processing Elements for matrix multiplications, allowing for pipelined execution. Experimental results demonstrate that FDHA achieves 3.28x speedup over an NVIDIA A100 GPU and 2.62x speedup over a SoTA diffusion accelerator.

ABSTRACT. Vision Transformers (ViTs) exhibit superior performance in computer vision tasks but face deployment challenges on resource-constrained devices due to high computational/memory demands. While Mixture-of-Experts Vision Transformers (MoE-ViTs) mitigate this through scalable architecture with sub-linear computational growth, their hardware implementation on FPGAs remains constrained by resource limitations. This paper proposes a novel accelerator for efficiently implementing quantized MoE models on FPGAs through two key innovations: (1) A dual-stage quantization scheme combining precision-preserving complex quantizers with hardware-friendly simplified quantizers via scale reparameterization, with only 0.28\% accuracy loss compared to full precision; (2) A resource-aware accelerator architecture featuring latency-optimized streaming attention kernels and reusable linear operators, effectively balancing performance and resource consumption. Experimental results demonstrate that our accelerator achieves nearly 155 frames per second, a 5.35x improvement in throughput, and over 80\% energy reduction compared to state-of-the-art (SOTA) FPGA MoE accelerators, while maintaining <1\% accuracy loss across vision benchmarks. The code will be open-sourced.

ABSTRACT. This paper proposes SkipNZ, a novel approach to reduce computational demands with negligible accuracy loss, improving the speed of sparse tensor convolutional neural networks. SkipNZ extends existing zero-value skipping techniques and enables skipping MAC operations for non-zero values. The evaluation results demonstrate that the proposed technique reduces computational load and improves overall performance with negligible accuracy loss. Compared to baseline, SkipNZ reduces execution time to 0.71× in AlexNet with Gap9 with 0.1% accuracy loss. Also, VGG16 with Gap8 achieves the reduction to 0.78× with 0.0% accuracy loss. The reduction in execution time comes from filtering out unnecessary MAC operations introduced by SkipNZ. The synthesized result shows that the additional hardware required by SkipNZ is only increased by 0.005% compared to the baseline.

ABSTRACT. Multi-DNN accelerators enable the simultaneous execution of multiple DNN workloads, enhancing performance by overlapping computations and memory access of multiple DNN workloads. However, this increases the demand on on-chip memory, which must handle the footprints of several workloads. Batching allows multiple consecutive DNN inferences from the same model to share weights, improving weight reuse and reducing off-chip access costs over a batch. However, for batching to be effective, input feature maps of all inferences in the batch must fit in on-chip memory. Thus, the offline-determined batch size of each workload can exhaust available on-chip memory when multiple workloads share it, resulting in reduced performance.

This paper introduces BATCH-DNN, a dynamic method for adapting batch size at a layer-by-layer basis to available on-chip memory, enhancing multi-DNN accelerator performance. It employs two techniques: adaptive cascaded sub-batching and adaptive sub-batch merging. Offline profiling assesses the footprint, while run-time adjustment establishes the maximum batch size on a layer-by-layer basis based on available on-chip memory.

ABSTRACT. Training large-scale distributed machine learning models imposes considerable demands on network infrastructure, often resulting in sudden traffic spikes that lead to congestion, increased latency, and reduced throughput—ultimately affecting convergence times and overall training performance. While gradient compression techniques are commonly employed to alleviate network load, they frequently compromise model accuracy due to the loss of gradient information.

This paper introduces NetSenseML, a novel network adaptive distributed deep learning framework that dynamically adjusts quantization, pruning, and compression strategies in response to real-time network conditions. By actively monitoring network performance, NetSenseML optimizes the balance between reducing data payloads and preserving model accuracy, minimizing the trade-offs between transmission efficiency, convergence speed, and performance.

Our approach ensures efficient resource usage by adapting reduction techniques based on current network conditions, leading to shorter convergence times and improved training efficiency. We present the design of the NetSenseML adaptive data reduction function and experimental evaluations show that NetSenseML can improve training throughput by a factor of 1.55 to 9.84× compared to state-of-the-art compression-enabled systems for representative DDL training jobs in bandwidth-constrained conditions.

ABSTRACT. Radio telescopes produce enormous amounts of data. Many of them use GPU clusters to combine the digitized antenna signals, usually in real time. Achieving high data rates is challenging: the PCIe bandwidth of discrete GPUs is limited, and without RDMA, handling 200 or 400~Gb/s Ethernet packets with telescope data is difficult.

The NVIDIA Grace Hopper is a novel, innovative system that eliminates the I/O bottleneck of traditional, discrete GPUs by using NVLink instead of PCIe. This opens the door to higher data rates, but faster hardware alone is not enough. In this paper, we combine hardware and software innovations to process Ethernet packets at no less than 1.2~Tb/s, a huge improvement over what was previously possible. We use the Data Plane Development Kit to minimize the receive overhead, and use a new feature that allows packet processing directly by the GPU. We demonstrate the data handling in a correlator application, analyze the performance, and show how to reduce the energy use.

The presented innovations enable the use of GPUs for more powerful telescopes with much higher data rates. The results are also of interest to (GPU) applications from other application domains with high I/O demands, especially if RDMA is not available.

ABSTRACT. Disaggregated memory separates compute and storage nodes into two independent memory pools, connected via RDMA or CXL links. Disaggregated memory improves resource utilization, saves cost overhead, and ensures elastic scalability of memory and compute resources. Tree indexes are essential guarantees in storage systems, such as databases or KV storage. Existing disaggregated memory systems suffer from poor write performance, mainly due to concurrency conflicts, frequent Structure Modify Operation(SMO) operations, and high lock overhead on tree index.

To solve this problem, we propose GECKO, a write-optimized Adaptive Radix Tree(ART) index structure for disaggregated memory. We leverage 1) a write-optimized buffer node to handle concurrent writes, improving write performance, 2) a threshold-based splitting strategy to reduce splits and optimize SMO operations, 3) a post-insertion lock design to reduce lock overhead and reduce insertion tail latency. We compare GECKO with state-of-the-art solutions, experiments show that we improves throughput by 1.43×-3.21× under write workloads while SMO operation time is reduced by 88.5%, and lock time is reduced by 87.9%.

ABSTRACT. Heterogeneous multi-node clusters with accelerators, such as GPUs, are increasingly the standard for HPC, where the MPI+OpenMP approach is commonly used for application development. However, this approach poses significant challenges for developers, especially in managing communication, synchronization, and load balancing across distributed nodes and accelerators. To address these challenges, this paper proposes the MPI Proxy Plugin (MPP), an extension of the LLVM OpenMP Offloading runtime that transparently offloads OpenMP target regions to remote accelerators via MPI. By abstracting communication and using the asynchronous mechanisms of MPI with C++20 coroutines, MPP provides a scalable alternative to MPI+OpenMP, enabling simpler development of heterogeneous HPC applications through the familiar OpenMP programming model. Experimental results shows that MPP achieves excellent scalability. For compute-intensive proxy-application, it scales nearly linearly, reaching a 63x speedup from 1 to 64 GPUs. While naive data transfers can degrade performance, this research reveals that extending OpenMP Target with collective operations (e.g., multi-device broadcast) simplifies development and improves performance, achieving up to 7x speedup in communication-bound benchmarks.

ABSTRACT. In-depth understanding of collective MPI communication is a major challenge for both beginners and experienced developers. It is challenging to grasp the operations of collective algorithms, as many performance analysis tools cannot break down collective operations into their underlying point-to-point communication. However, collective communication is a key factor in optimizing the performance of parallel programs. EduMPI is a novel tool for parallel programming education, providing near-real-time visualization of collective MPI algorithms. It displays per-process data exchange and highlights performance issues like Late Sender and Late Receiver. This paper introduces the visualization of collective communication in EduMPI, demonstrates how this representation aids in understanding the concept of collective communication and its relevance for performance optimization, and evaluates its effectiveness in an educational context. EduMPI bridges the gap in understanding complex collective MPI operations by providing a transparent view of the underlying processes, enabling students to visualize and better understand the communication flow.

ABSTRACT. Compute Express Link (CXL) is recognized as a revolutionary technology in high-performance computing (HPC) system design, driven by the growing demand for efficient and scalable memory solutions tailored to memory-centric workloads. However, despite its potential, evaluating CXL performance in real-world scenarios is challenging due to the lack of CXL hardware and the high costs of building a large-scale distributed system. To address this, we propose TSim4CXL, a novel trace-driven simulation framework for CXL-based HPC systems that provides accurate timing simulations within a practical timeframe. TSim4CXL separates computing resources from the CXL memory system, generating traces and simulating the memory system using SystemC’s discrete-event modeling. By modeling the CXL interconnect at the protocol level with various configuration parameters, TSim4CXL allows us to explore the design space of HPC architecture. The accuracy of our CXL simulation model is validated using CXL hardware provided by Samsung Electronics. First, we compare load latencies using a custom microbenchmark on CXL hardware with simulation results and adjust the CXL parameters in our simulator accordingly. Second, we assess communication latency by running LAMMPS applications, ensuring the simulation results align with real-world performance. In addition, we perform design space exploration with two memory-centric applications, up to 25 CPU nodes for LAMMPS and 4 GPU nodes for LLM training. Furthermore, we compare the performance of target applications by executing multiple DRAM simulators, demonstrating how the memory bandwidth affects simulated time. These experiments prove the viability of the proposed simulation framework.

ABSTRACT. As we reach exascale, production High Performance Computing (HPC) systems are increasing in complexity. These systems now comprise multiple heterogeneous computing components (CPUs and GPUs) utilized through diverse, often vendor-specific programming models. As application developers and programming models experts develop higher-level, portable programming models for these systems, debugging and performance optimization requires understanding how multiple programming models stacked on top of each other interact with one another. This paper discusses THAPI (Tracing Heterogeneous APIs), a portable, programming model-centric tracing framework: by capturing comprehensive API call details across layers of the HPC software stack, THAPI enables fine-grained understanding and analysis of how applications interact with programming models and  heterogeneous hardware. Leveraging state of the art tracing framework like the Linux Trace Toolkit Next Generation (LTTng) and tracing much more than other tracing toolkits, focused on function names and timestamps, this approach enables us to diagnose performance bottlenecks across the software stack, optimize application behavior, and debug programming model implementation issues.

ABSTRACT. The data-parallel distributed training technique is a de facto approach in training large-scale deep neural networks (DNNs) using synchronous stochastic gradient descent (S-SGD). However, S-SGD requires iteratively aggregating the distributed gradients through an AllReduce collective, which easily results in significant data communication across distributed GPUs and thus limits the scaling efficiency of the training system. In this paper, we propose an efficient and practical gradient sparsification and quantization algorithm, named SQ-DeAR, which not only significantly reduces the communication traffic, but also allows overlapping communications with both feed-forward and backpropagation computations to further improve the training performance. In addition, to improve the computation efficiency of gradient sparsification, we design a batched gradient sparsification to reduce the number of GPU launches. Performance evaluation on a 32-GPU cluster shows that SQ-DeAR outperforms state-of-the-art solutions by 1.17x-7.0x.

ABSTRACT. Multi-Agent Reinforcement Learning (MARL) enables multiple autonomous agents to simultaneously learn and make decisions in complex, interactive environments. Among various MARL paradigms, Independent Learning (IL) remains a dominant approach due to its simplicity and scalability, where each agent optimizes its policy independently, treating others as part of the environment. While IL eliminates the need for explicit inter-agent communication, existing MARL implementations fail to exploit its inherent parallelism. Current implementations train agents sequentially on a single accelerator, leading to severe underutilization of modern compute resources, particularly on multi-GPU platforms. In this work, we propose a multi-GPU training scheme that efficiently distributes independent agent policies across compute devices without altering the original IL semantics. To further enhance scalability, we design a dynamic load-balancing strategy that adaptively assigns training workloads based on computational demands and the varying capabilities of different GPUs, ensuring efficient utilization of hardware resources. Our approach achieves up to 15.5x higher throughput than state-of-the-art MARL implementations, demonstrating that fully leveraging the parallelism of IL can significantly accelerate MARL training, opening new possibilities for large-scale multi-agent learning in high-dimensional environments. We open-source our work with optimized implementations of widely used independent learning algorithms, enabling scalable MARL training on diverse accelerator platforms.

ABSTRACT. Large language models~(LLMs) are known for their high demand on computing resources and memory due to their substantial model size, which leads to inefficient inference on moderate GPU systems. Techniques like quantization or pruning can shrink model sizes but often impair accuracy, making them unsuitable for practical applications. In this work, we introduce \modelname{}, a high-performance inference engine designed to speed up LLM inference without compromising model accuracy. \modelname{} incorporates three innovative methods to increase inference efficiency: 1) model partitioning to allow asynchronous processing of tasks across CPU computation, GPU computation, and CPU-GPU communication, 2) an adaptive partition algorithm to optimize the use of CPU, GPU, and PCIe communication capabilities, and 3) a token assignment strategy to handle diverse prompt and generation tasks during LLM inference. Comprehensive experiments were conducted with various LLMs such as Mixtral, LLaMA-2, Qwen, and PhiMoE across three test environments featuring different CPUs and GPUs. The experimental findings demonstrate that \modelname{} achieves speeds between $1.11\times$ to $1.80\times$ faster in decoding and $1.69\times$ to $6.33\times$ faster in pre-filling, leading to an overall speedup ranging from $1.25\times$ to $2.04\times$ compared to state-of-the-art solutions, llama.cpp and Fiddler.

ABSTRACT. Emerging supercomputers increasingly rely on heterogeneous nodes composed of GPUs, many-core CPUs, and multiple NUMA regions. Efficient data movement across these components is critical to achieving high performance. In this talk, I will present benchmark-driven analysis of emerging heterogeneous systems, highlighting the cost of data transfers both within and between nodes. Based on these insights, I will present architecture-aware data movement optimizations that accelerate key operations, including the all-reduce. I will also discuss how these optimizations generalize to collective communication libraries (CCLs), highlighting opportunities for interoperability between MPI and CCLs.

ABSTRACT. This presentation discusses omnitrees, a novel data structure that generalizes traditional octrees to support anisotropic refinement in adaptive mesh refinement (AMR) schemes. This addresses a long-standing inefficiency in AMR for intrinsically anisotropic problems: conventionally, much resolution is spent where little information is gained. Omnitrees directly address this problem by allowing to selectively refine only the most informative dimensions, which increases the order of convergence in AMR schemes. We validate this on the problem of shape representation across 4,166 three-dimensional objects. The result are efficient data structures, insights into anisotropy, convergence, and information density, and of course, cat GIFs!

ABSTRACT. Software performance evolves over time, yet traditional debugging and profiling remain reactive, costly, and disconnected from development workflows. Performance drift—gradual degradation in execution efficiency due to code modifications—often goes undetected until it causes significant slowdowns, forcing late-stage debugging and costly fixes. This talk presents a vision for AI/ML-driven proactive performance-drift detection, where models continuously monitor software evolution, identifying inefficiencies before they degrade execution. By combining static analysis (abstract syntax trees) with dynamic insights from nightly tests, this framework enables early detection of performance-impacting changes. Traditional ML approaches require full model retraining whenever code changes or new runtime data become available, making them impractical for fast-moving development cycles. Few-shot learning eliminates this overhead by allowing models to update incrementally with minimal new data. Attention-based representation learning further enhances interpretability by prioritizing performance-critical features, enabling more targeted interventions. This framework supports two key decision-making processes where (1) developers can receive automated feedback on whether a code change improves or degrades performance, enabling early intervention; (2) the insights can guide hardware configuration choices and runtime parameter tuning. This approach can be seamlessly integrated into CI/CD pipelines to achieve software that not only remains correct but also maintains efficiency as it evolves.

ABSTRACT. DAPSO is a GPU--accelerated, asynchronous variant of Particle Swarm Optimization (PSO) for Big Data on Apache Spark, addressing the limitations of traditional PSO. Compared to a standard version that uses RDDs (DSPSO), it uses the RAPIDS acceleratorand a new SuperRDD abstraction to allow finer-grained asynchrony and parallelism. Experiments in energy consumption prediction and tumour classification show speeds up to 4 times faster, strong scalability across particle number, iterations and dataset size.The predictive accuracy achieved is on a par with that of DSPSO, illustrating that GPU integration is an effective enhancement for metaheuristics in distributed systems.

ABSTRACT. Hardware accelerators are essential for efficiently executing compute- and data-intensive algorithms such as convolutional neural networks. The V2PRO is a programmable, massive-parallel vertical vector processor architecture designed for both configurability and scalability. To illustrate its adaptability under different resource and performance constraints, we present two FPGA-based demonstrators: one tailored for maximum performance on an UltraScale+ Xilinx FPGA (TySOM-3A-ZU19EG Board) and another optimized for a low-cost Zynq-7000 Xilinx FPGA (PYNQ-Z1 Board).

ABSTRACT. Hyperledger Fabric has established itself as a prominent permissioned blockchain framework for enterprise applications, offering modular architecture, pluggable consensus mechanisms, and privacy-preserving features. Despite these advantages, the platform faces challenges in transaction throughput and rejection rates, particularly in high-demand scenarios. These limitations stem from Fabric's transaction flow architecture, which involves endorsement, ordering, and validation phases that can create bottlenecks under heavy loads.

This research proposes a novel approach to enhance Fabric's performance through a transaction dependency-aware execution mechanism. Our modifications introduce: (1) a dependency flagging system during endorsement where transactions are marked as independent (0) or dependent (1), utilizing a hashmap at the leader endorser to track transaction dependencies; (2) optimized block construction at the ordering service that prioritizes independent transactions; (3) creation of a Directed Acyclic Graph (DAG) representing transaction dependencies within each block; and (4) parallel execution of independent transactions at the committer level while processing dependent transactions according to their DAG relationships. Experimental results demonstrate improvements in transaction throughput and reduced rejection rates compared to standard Fabric implementation. Our findings suggest that dependency-aware transaction processing represents a promising approach to scaling Hyperledger Fabric for enterprise applications with complex transaction patterns.

ABSTRACT. The main issue with the measurement of energy consumption of any kind of algorithm is establishing a methodology that allows an actionable comparison between different implementations in systems that actively implement their own energy-optimization strategies. In this poster, we will try to observe the effects of these strategies in energy-measurement experiments by looking at how the sequence or moment at which a specific measurement is taken affects the results. We will then propose statistics summaries that might be more actionable than a simple average.

ABSTRACT. The paper is devoted to the problem of the automated construction of parallel programs. In the paper ParSolGen (Parallel Solvers Generator) - a tool that automatically constructs a highly efficient parallel program from a given numerical algorithm description is presented. It classifies the input numerical algorithm description written in ParSolGen language, determines the class it potentially belongs to and constructs the parallel program using the corresponding domain-specific parallel program construction module. This enables ParSolGen to employ well-known manual parallel programming techniques in the process of the automated parallel program construction. This approach improves the performance of the automatically generated parallel programs, as confirmed by performance tests.

ABSTRACT. Data centres have close correlations between IT systems and the required infrastructure components, which are too complex to crack using classical design concepts. The project IT-Zauber tackles this gap by utilising a digital twin of the data centre, which comprises numerous models for the assorted systems and components. In this work, we present our ongoing modelling efforts for the LZR data centre of the TU Dresden.

ABSTRACT. Federated Learning (FL) enables collaborative model training across distributed clients while preserving data privacy. Traditional FL techniques have used a centralized server for learning. However, a centralized server suffers from bottlenecks and single-point failures due to the central server. A natural way to address this problem is to have a decentralized FL which does not have any central server. Decentralized Learning Systems can be broadly classified as synchronous and asynchronous. In Synchronous FL systems all workers compute gradients and wait for each other before aggregating updates, ensuring consistency. But in real-life distributed systems, device and network variations make synchronization impractical. Thus asynchronous FL is a practical system where clients learn independently without waiting for each other. In this work, our research is conducted in two phases to handle these issues. (a) In Phase 1, In Phase 1, we address the challenges in asynchronous systems by developing a FL framework that allows clients to update independently. (b) In Phase 2, we incorporate fault tolerance mechanisms into the asynchronous FL setup to manage issues such as client failures and message drops, ensuring robust performance and convergence under varying conditions.

ABSTRACT. Training today's hundred-billion-parameter neural networks is limited less by model architecture than by the efficiency with which accelerator clusters are utilized. Existing auto-parallel planners optimize at most three out of four major axes-data (DP), tensor (TP), pipeline (PP) and optimizer (OP) parallelism-and rarely co-tune micro-batch size or activation recomputation, leaving significant speedups untapped. We present H2O, a two-level, holistic hyper-parameter optimizer that jointly explores DP–TP–PP–OP degrees together with micro-batch size, stage assignment and selective recomputation under a unified analytic compute-communication-memory model. A coarse Level 1 search prunes the vast design space, while a fine-grained Level 2 search balances pipeline stages and recomputation to minimize overall iteration time under device-memory constraints. On a 128 devices cluster training a 141-billion-parameter DeepSeek model, H2O shows 22.38% faster than an expert hand-tuned baseline and 36.72% faster than the state-of-the-art auto-parallel planner. These results demonstrate that cross-axis optimization is already critical for contemporary hardware and will be indispensable in the trillion-parameter era.

ABSTRACT. Hardware overprovisioning is a widely used technique to improve the average power utilization of computing systems by capping the processor’s power consumption. However, applying uniform power caps across sockets in multiprocessor systems can degrade the performance of co-running applications due to workload variation. Existing solutions primarily aim to enhance power utilization and system throughput but ignore fairness in sharing surplus power between applications and rely on processor-selected frequency settings under a power cap, which may be suboptimal. This thesis addresses these limitations through a two-stage power management approach. First, it proposes a system that dynamically and bidirectionally redistributes surplus power among applications to improve overall system throughput while ensuring fairness. When it identifies an application with surplus power, it donates power to other applications exhibiting high CPU utilization. When donor applications transition to the high CPU utilization phase, they are rewarded by having a portion of the transferred power returned to them. Second, it improves performance by configuring optimal frequency settings for each application under a power cap rather than relying on default processor settings. Once the performance is optimized, the system further reallocates power to balance performance gains across applications to improve the fairness between applications. The proposed system is evaluated on a quad-socket, 72-core Intel Xeon server using diverse HPC application mixes and power cap settings. The results demonstrate significant improvements in both system throughput and application-level fairness.

ABSTRACT. Gate sizing is important in VLSI design to optimize performance and meet timing constraints. Multi-core CPU-based approaches have been widely used to speed up the gate sizing algorithm, but their scalability is typically limited to 8-16 threads. To address this limitation, we propose a GPU algorithm to accelerate a time-consuming routine of gate sizing, namely the library cell (libcell) selection process, in an industrial-standard sizer. By leveraging both block- and warp-level parallelism, our algorithm can greatly accelerate the libcell selection time. Experimental results show that our GPU implementations achieve up to 38.13x speedup over a 16-core CPU baseline, while warp-level sizing can further achieve additional 4.77% improvement over block-level sizing.

ABSTRACT. Modern machine learning (ML) pipelines increasingly suffer from data preprocessing bottlenecks-especially in scientific workloads using structured formats like HDF5. Existing systems lack support for remote preprocessing, provide no scheduling across heterogeneous environments, and underutilize cloud resources. We introduce SCOPE (Smart Computational Offloading for Preprocessing Efficiency), a runtime system that enables dynamic, cost-aware partitioning of preprocessing tasks between cloud and local heterogeneous sytem. SCOPE extends the h5py API to support remote transformations and uses a scheduler to select the optimal offload point to minimize end-to-end time. It also leverages idle cloud cycles to precompute samples ready to serve, improving throughput and reducing latency. Evaluations on CIFAR-10 and ImageNet-1K with ResNet models show that SCOPE significantly reduces preprocessing overhead and accelerates training, enabling scalable and efficient ML in cloud-connected environments.

ABSTRACT. This PhD project addresses the automated generation of high-performance, portable code for dynamic programming problems, with a focus on RNA secondary structure prediction. The work combines polyhedral model-based compilation, domain-specific knowledge, and large language models (LLMs) to produce cache-efficient, parallel implementations targeting heterogeneous platforms including multicore CPUs and GPUs.

The research evolved through three stages. First, prompt-based LLMs (e.g., GPT-3.5) were applied to transform OpenMP code into CUDA, successfully generating correct and compilable kernels for complex algorithms such as Nussinov’s, without manual intervention. Next, a hybrid method combining polyhedral schedules with lightweight GPU tuning achieved up to $40\times$ speedup over GPT-based generation for Nussinov, with architecture-specific adjustments. Finally, a dedicated model named \textit{omniCUDA} was fine-tuned using synthetic and handcrafted OpenMP–CUDA pairs. This fully automated solution outperforms baseline PolyBench implementations and shows that aligned training on input–output code pairs enables high-quality source-to-source translation.

This work presents a semi-automated, retargetable compiler pipeline for affine dynamic programming kernels, capable of generating optimized code for architectures including x86, RISC-V, and CUDA with minimal manual intervention. By integrating classical compiler theory with modern AI-based techniques, it bridges the gap between high-level algorithm specifications and architecture-aware implementations across heterogeneous platforms.

ABSTRACT. Oversubscription is a popular method of combatting the negative effects of virtual machine underutilisation in cloud datacenters, primarily increased energy consumption. It has been observed that the cores within a virtual machine are not underutilised to the same degree. With this knowledge, CoreWaterfall was developed to allow the individual cores of virtual machines to be oversubscribed by different amounts. By employing this core-specific approach to allocation and scheduling of virtual machines, the energy consumption of a datacenter is reduced when compared to a standard flat oversubscription mechanism.

ABSTRACT. As parallel computers continue to grow larger and parallel programs become increasingly complex, traditional block-synchronous programming approaches reach their limits. Asynchronous paradigms, such as tasking and offloading, have been shown to yield significant performance improvements for suitable applications, particularly those with irregular, dynamic workloads. While asynchronous paradigms offer a lot more flexibility during programming, they also make the performance analysis of applications using them more complex. This results in a high need for easy-to-use, scalable performance analysis tools. While tool support for tasking and offloading exists, these solutions often work post-mortem, and may not scale well to high numbers of tasks. Additionally, explicit support for offloading is predominantly post-mortem and rarely integrates the analysis of dependencies and inter-process communication. The PhD work discussed in this paper aims to investigate the possibilities of on-the-fly performance analysis of tasking and offloading applications. It will first define new possible metrics to be tracked alongside an application's execution. Extending an existing on-the-fly performance analysis tool will evaluate the metrics in terms of their insightfulness for performance analysis. Additionally, a performance analysis workflow with the extended tool will be developed.

ABSTRACT. With large-scale pre-trained models achieving breakthrough progress in numerous domains, the computational complexity and memory demands during training and inference phases present increasingly severe challenges. The Tianhe new-generation supercomputer, leveraging their formidable computational power and unique architectural advantages, offers new opportunities for efficient large model training and inference. Targeting these characteristics, this paper investigates and proposes hardware-software co-optimization strategies aimed at enhancing the efficiency of large model training and inference on the Tianhe platform. By establishing a deep co-optimization mechanism between hardware features and model architecture, we achieve key objectives including maximizing data locality under extremely low memory bandwidth constraints, optimizing the utilization of heterogeneous many-core resources, and balancing communication overhead with computational efficiency in distributed strategies. Experimental results demonstrate that the proposed optimization framework achieves significant performance improvements on typical large models, providing an effective solution for industrial-scale large model applications on the Tianhe supercomputing platform.

ABSTRACT. The growth of heterogeneous multi-core processors and the irregular nature of scientific computing have led to the increased use of parallel decomposition strategies. However, developing high-performance parallel applications remains challenging due to technical complexity and time constraints. Task graph computing systems facilitate the implementation of parallel strategies on heterogeneous platforms. Existing systems face three key limitations: (1) limited support for in-graph control flow, which restrict task graph expressiveness; (2) inability to overlap task execution, leading to resource bottlenecks; (3) poor cross-hardware portability, hindering adaptation to emerging architectures. To address these issues, we propose DCG-DDQ, a cross-platform task graph computing system. DCG-DDQ enhances task graph expressiveness by using directed cyclic graphs to embed in-graph control flow. A finite state machine model addresses complex scheduling challenges in cyclic task graphs. By decoupling application logic from hardware-specific code, DCG-DDQ ensures portability across heterogeneous architectures. Experimental results show that DCG-DDQ outperforms OpenMP in execution time for irregular task scheduling.

ABSTRACT. Dedicated hardware accelerators are essential to meet the compute and power demands of data-intensive applications such as convolutional neural networks. Thereby, the design of the accelerator's memory system is critical: parallel accelerators often rely on DMA controllers instead of traditional cache hierarchies to match massive on-chip compute throughput. However, when scaling the architecture, the DMA count grows, and contention at the external memory controller becomes a serious bottleneck. Moreover, different memory technologies - from single-channel DDR4 to multi-channel HBM - demand different requirements on the accelerator’s memory architecture. This paper presents an ongoing PhD research aimed at optimizing DMA-based memory systems through a novel Direct Cached Memory Access (DCMA). The DCMA is used to transparently optimize memory transfers, effectively hiding external memory complexity from the accelerator architecture. We describe the research methodology, report preliminary results from an FPGA-based evaluation, and outline the next steps.

ABSTRACT. Counting queries are fundamental operations in many machine learning applications. In this work, we introduce the ccNUMA-aware strategy for the parallel execution of a counting query stream. Our approach improves data locality and ensures balanced workloads across NUMA domains. The proposed strategy is compatible with any underlying counting method, including our previously developed auto-scheduling mechanism that selects the optimal method for each query in an online manner. Experiments on two AMD-based dual-socket servers show up to 10× speedup over a standard parallelization approach for large-scale query workloads.

ABSTRACT. 3D Gaussian Splatting (3DGS) enables high-fidelity real-time rendering but remains underexplored on embedded hardware. This work benchmarks the baseline 3DGS pipeline across heterogeneous systems, including NVIDIA Jetson Xavier, Orin, and RTX 4090. We evaluate rendering speed, energy-per-pixel efficiency, and visual quality across four datasets. Results reveal trade-offs between performance and energy, highlighting each platform’s deployment suitability. Findings are contextualized within energy trends in embedded VISLAM systems. This study offers a reference point for constrained neural rendering and motivates future integration of SLAM and 3DGS pipelines for real-time robotic applications.

ABSTRACT. Code generation is a critical stage in compiler design, responsible for translating high-level intermediate representations into optimized machine code. This paper presents a systematic exploration of code generation within the LLVM (Low-Level Virtual Machine) compiler infrastructure, emphasizing its modular backend design and target-independent optimizations. We investigate the transformation pipeline from LLVM Intermediate Representation (IR) to target-specific assembly, highlighting the roles of instruction selection, register allocation, and machine-specific lowering. The paper introduces enhancements to the LLVM code generation process, including custom pattern matching for improved instruction selection and a novel register allocation heuristic that minimizes spill code in irregular architectures. Our results reaffirm LLVM's suitability as a research platform for backend innovations and contribute new insights into scalable and portable code generation strategies.

ABSTRACT. In this paper we present work already done in the scope of the PhD thesis related to selection of the best infrastructure for a wide range of medical applications. In the beginning we define motivation for the work. Later we describe classes of those applications highlighting most important resources being used by each class. It is followed by the state of the art analysis of appropriate monitoring platforms. Later we present a set of experiments that were already performed as well as those that are planned for the remaining duration of the work on the thesis. We focus on large-scale systems such as HPC clusters as well as computational clouds. We compare and contrast various aspects of those infrastructures, highlighting features most critical for each class of applications. Finally, the paper provides a summary of the results gathered, followed by a discussion of their impact on the subjects of the work.

ABSTRACT. Serverless Computing (FaaS) has become a popular paradigm for deep learning inference due to the ease of deployment and pay-per-use benefits. However, current serverless inference platforms encounter the coarse-grained and static GPU resource allocation problems during scaling, which leads to high costs and Service Level Objective (SLO) violations in fluctuating workloads. Meanwhile, current platforms only support horizontal scaling for GPU inferences, thus the cold start problem further exacerbates the problems. In this paper, we propose HAS-GPU, an efficient Hybrid Auto-scaling Serverless architecture with fine-grained GPU allocation for deep learning inferences. HAS-GPU proposes an agile scheduler capable of allocating GPU Streaming Multiprocessor (SM) partitions and time quotas with arbitrary granularity and enables significant vertical quota scalability at runtime. To resolve performance uncertainty introduced by massive fine-grained resource configuration spaces, we propose the Resource-aware Performance Predictor (RaPP). Furthermore, we present an adaptive hybrid auto-scaling algorithm with both horizontal and vertical scaling to ensure inference SLOs and minimize GPU costs. The experiments demonstrated that compared to the mainstream serverless inference platform, HAS-GPU reduces function costs by an average of 10.8x with better SLO guarantees. Compared to state-of-the-art spatio-temporal GPU sharing serverless framework, HAS-GPU reduces function SLO violation by 4.8x and cost by 1.72x on average.

ABSTRACT. The serverless computing model offers users flexible, pay-as-you-go services. However, existing frameworks face challenges such as resource over-subscription and workload over-scaling when deploying graph processing jobs in serverless environments. To address these limitations, we introduce CGP–Graphless, which utilizes CPU–GPU heterogeneous computing to enable efficient vertical scaling. This approach divides graph processing into two phases: querying on a core proxy graph and correction on the full graph. GPU containers are allocated for the querying phase, while CPU containers handle the correction phase. Furthermore, we propose an adaptive pipelined scheduling strategy between these phases, which leverages pressure-aware intra-pipeline scaling to transform excessive horizontal scaling into vertical scaling, thereby achieving efficient serverless graph computation. Experiments show that CGP–Graphless improves end-to-end performance by up to 2.00x over FaaSGraph under concurrent stress evaluations, while halving allocated CPU cores through additional GPU containers. In short-interval query scenarios, CGP–Graphless further reduces average request latency by 3.30x compared to FaaSGraph.

ABSTRACT. With the rapid advancement of Artificial Intelligence (AI) and High-Performance Computing (HPC), GPU-centric clusters have become pivotal in driving research across diverse disciplines. Consequently, campus data centers are increasingly aggregating substantial computing resources. However, akin to commercial GPU and AI clusters, campus GPU clusters frequently encounter challenges related to GPU underutilization.

While pooling technologies offer a viable solution to this issue, existing GPU pooling approaches fall short in effectively supporting environments where multiple applications coexist within campus data centers.

This paper presents gPooling, a novel pooling scheme that leverages device driver hijacking to optimize GPU resource allocation. We designed a benchmark based on real-world traces from a campus data center and deployed gPooling within a GPU cluster environment. Experimental results from both benchmarking and actual deployment demonstrate that gPooling significantly enhances GPU utilization and reduces user waiting times, thereby improving the overall efficiency of campus GPU clusters.

ABSTRACT. Embedding-based recommender systems (RecSys) face critical bottlenecks in storing massive embedding tables (EMTs) and accelerating latency-sensitive lookup operations. While serverless computing and disaggregated memory architectures offer resource efficiency, existing solutions struggle with cold-start penalties, state transfer overheads, and rigid resource scaling for EMT-bound workloads. We present ServerlessRec, a serverless framework that integrates memory disaggregation with kernel-space RDMA-based remote memory mapping (rmap) and remote fork (rfork) to enable elastic EMT lookups. ServerlessRec decouples compute nodes (CNs) from memory nodes (MNs), dynamically autoscaling serverless functions on CNs for bursty lookups while storing EMTs on MNs. ServerlessRec improves latency-bounded throughput by 3.8× over DisaggRec, reduces resource waste by 62% vs. ElasticRec, which demonstrates how kernel-level memory disaggregation unlocks serverless advantages for EMT-bound applications.

ABSTRACT. Despite recent advancements in data stream processing frameworks that emphasize the importance of real-time data handling, scalability remains a significant challenge, directly impacting both throughput and latency. Although prior research has investigated this issue on local machines and cloud clusters, there remains a gap in studies exploring its implications for modern high performance computing (HPC) systems, primarily due to the scarcity of scalable measurement tools. This work introduces SProBench, a novel benchmark designed to evaluate scaling performance of data stream processing frameworks on HPC systems, incorporating a modular architecture and various workload pipelines. Notably, this benchmark suite is, to the best of our knowledge, the first to provide native support for SLURM-based clusters. The current implementation of SProBench enables out-of-the-box performance evaluation for Apache Flink, Apache Spark Streaming, and Apache Kafka Streams, while allowing for easy integration of additional frameworks with minor modifications. Building upon established best practices from previous research, SProBench is designed for effective execution on HPC systems and is highly scalable. Real-world cluster evaluations denote the benchmarks linear scaling capabilities. Furthermore, this study outlines the collection and integration of metrics from external monitoring tools for post-processing analysis. This data confirmed and validated the benchmarks functionality and efficient scaling performance.

ABSTRACT. Sequence alignment is a crucial step in next-generation sequencing data analysis. However, most sequence aligners face performance challenges due to high computational complexity and extensive random memory access patterns, making them a significant bottleneck in the overall analysis pipeline, such as the industry gold standard BWA-MEM. The next-generation Sunway platform, with its high computational power and unique heterogeneous architecture, presents new opportunities for enhancing the efficiency of sequence alignment. In this work, we introduce SWBWA, a high-accuracy and high-performance sequence aligner designed for the new Sunway architecture. By redesigning the parallel framework tailored for Sunway, performing software prefetching optimization, vectorizing the striped Smith-Waterman algorithm, and addressing memory access bottlenecks in bigshare mode, SWBWA achieves a 330× speedup over the single-threaded unoptimized version. Additionally, SWBWA running on a Sunway workstation can achieve 1.2–1.4× speedups compared to BWA-MEM running on a dual-socket 48-core x86 server, while ensuring nearly identical output. The source code is publicly available at https://github.com/RabbitBio/SWBWA.

ABSTRACT. Analyzing gigapixel images is recognized as computationally demanding. In this paper, we introduce PyramidAI, a technique for analyzing gigapixel images with reduced computational cost. The proposed approach adopts a gradual analysis of the image, beginning with lower resolutions and progressively concentrating on regions of interest for detailed examination at higher resolutions. We investigated two strategies for tuning the accuracy-computation performance trade-off when implementing the adaptive resolution selection, validated against the Camelyon 16 dataset of biomedical images. Our results demonstrate that PyramidAI substantially decreases the amount of processed data required for analysis by up to 2.65×, while preserving the accuracy in identifying relevant sections on a single computer. To ensure democratization of gigapixel image analysis, we evaluated the potential to use mainstream computers to perform the computation by exploiting the parallelism potential of the approach. Using a simulator, we estimated the best data distribution and load balancing algorithm according to the number of workers. The selected algorithms were implemented and highlighted the same conclusions in a real-world setting. Analysis time is reduced from more than an hour to a few minutes using 12 modest workers, offering a practical solution for efficient large-scale image analysis.

ABSTRACT. The molecular dynamics simulation code GROMACS is designed to run natively on CPUs and has supported GPU acceleration for computationally intensive electrostatic calculations using the Particle Mesh Ewald method since version 2018. Over the past few years, developers have further enhanced GROMACS by introducing GPU offloading for bonded interactions and the calculations of updates and constraints. These advancements raise important questions: Which hardware configuration delivers optimal performance? What are the best runtime parameters? How does performance vary across different hardware setups and software versions? To address these questions, extensive benchmarks have been conducted on various hardware architectures, including both CPUs and GPUs. The results provide valuable insights into scaling behavior, show performance increases with the most recent program version, and reveal benefits of newer GPU models. Additionally, the findings highlight the importance of evaluating runtime parameters on a case-by-case basis, tailored to the specific hardware and project.

ABSTRACT. The application of scale-resolving simulations such as LES and DNS remains challenging in numerous engineering applications due to the substantial computational demands and the limited available computational resources. It was thoroughly demonstrated in the last decade that high-order numerical methods show significant advantages in terms of computational cost required to achieve a given accuracy for DNS and LES and that they can be implemented efficiently on traditional CPU-based system architectures. However, the improvements in performance for CPU-based systems has slowed down significantly over the last years. Consequently, more specialised architectures, such as GPUs, have become increasingly popular in the last decade. These architectures still promise considerable performance improvements per generation and thus allow more complex flow cases to be tackled, while also yielding better performance per invested amount of energy. However, CPU codes cannot be easily run on GPU hardware, since they exhibit a distinctly different architecture and require different programming languages. In this talk it is demonstrated how existing codebases can be translated efficiently to GPU hardware, both in terms of implementation effort and computational performance. For this, the unstructured DG code FLEXI~\cite{Krais2019,Kempf2024} is utilized as an example. Finally, it is shown that FLEXI is capable of tackling challenging large-scale problems with complex geometries on modern HPC systems.

ABSTRACT. This talk presents the ongoing efforts to port Vistle – a software for creating immersive extended reality visualizations of scientific data – to the GPU by integrating the portable visualization toolkit Viskores. The approach is illustrated by sample computational fluid dynamics (CFD) visualizations of the light-house cases from the Center of Excellence in Exascale CFD (CEEC) project.

ABSTRACT. Simulating plasma turbulence is essential for understanding and optimizing the performance of fusion devices, but it poses significant computational challenges. A key difficulty lies in the intrinsically multiscale nature of turbulence, requiring one to resolve spatial scales ranging from the ion to the electron gyroradius, which are separated by a factor 60  in deuterium plasmas. To address this, we propose an in-situ coupling between GENE-X, a full-f global electromagnetic gyrokinetic code that resolves ion-scale turbulence, and GENE, used in the local (flux-tube) limit to model only electron-scale turbulence. Electron-scale fluxes computed by GENE are fed back into GENE-X as an effective heat source. A major challenge in this approach is the disparity in computational cost between the two turbulence models. We overcome this by introducing an in-situ detection mechanism within GENE-X to dynamically identify regions of interest and immediately trigger corresponding GENE simulations, without intermediate file I/O. This hybrid, scale-bridging strategy enables efficient and accurate modeling of multiscale turbulence, reducing computational overhead while preserving simulation fidelity.