科研成果详情

HyperAttention is a new attention mechanism proposed to address the inevitability of quadratic time complexity and the conditionality of linear time when dealing with extended contexts, which allows the algorithm to maintain the computational complexity in linear time despite having large entries or a large stability rank in the matrix. HyperAttention improves the speed of inference and training of the large language model, with only a slight performance degradation and a significant speedup compared to FlashAttention. In order to further improve the accuracy of HyperAttention while maintaining its speed, we optimised each module for the original algorithm to improve its efficiency. Our main contribution is to change the locally sensitive hash algorithm used in HyperAttention for detecting large values and instead use k-means for indexing, further reducing the model’s perplexity. In addition, we add applications on chatglm3-6b and baichuan-7b as well as chatglm3-6b-32k which supports long contexts in practice to make up for the lack of informativeness of HyperAttention in terms of practical applications, and test results are given. The experiments reveal that although the improved algorithm has a longer and relatively stable time complexity, it significantly reduces perplexity. Notably, with 10 to 15 replacement layers, the increase in time complexity is less than the reduction in perplexity, achieving a favourable balance between performance and efficiency. Moreover, we also analyse the feasibility of other sampling methods to improve accuracy and discuss the strategy for selecting the patch layers to apply the fast algorithm.