Presentation
Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
DescriptionThe adaptation of pre-trained LLMs to diverse downstream tasks through fine-tuning is essential for numerous applications. However, the inefficiency of parameter-efficient fine-tuning (PEFT) techniques presents significant challenges regarding time investments and operational costs. In this paper, we first introduce a nuanced form of sparsity, termed Shadowy Sparsity, which is distinctive in fine-tuning and has not been adequately addressed for acceleration. Under Shadowy Sparsity, we propose Long Exposure, an efficient system to accelerate PEFT for LLMs. Long Exposure comprises three key components: Shadowy-sparsity Exposer employs a prolonged sensing range to capture more sparsity details under shadowy sparsity; Sequence-oriented Predictor provides efficient yet accurate predictions to handle large-sequence inputs and constantly evolving parameters; and Dynamic-aware Operator facilitates more structured computational patterns and coalesced memory accesses to address dynamic sparse operations. Comprehensive evaluations demonstrate that Long Exposure outperforms state-of-the-arts with up to 2.49x speedup in end-to-end fine-tuning, offering promising advancements in PEFT acceleration.