Context Packing

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, Juanzi Li

Extending large language models to effectively handle long contexts requires instruction fine-tuning on input sequences of similar length. To address this, we present LongAlign -- a recipe of the instruction data, training, and evaluation for long context alignment. First, we construct a long instruction-following dataset using Self-Instruct. To ensure the data diversity, it covers a broad range of tasks from various long context sources. Second, we adopt the packing and sorted batching strategies to speed up supervised fine-tuning on data with varied length distributions. Additionally, we develop a loss weighting method to balance the contribution to the loss across different sequences during packing training. Third, we introduce the LongBench-Chat benchmark for evaluating instruction-following capabilities on queries of 10k-100k in length. Experiments show that LongAlign outperforms existing recipes for LLMs in long context tasks by up to 30\%, while also maintaining their proficiency in handling short, generic tasks. The code, data, and long-aligned models are open-sourced at https://github.com/THUDM/LongAlign.

Lvmin Zhang, Shengqu Cai, Muyang Li, Gordon Wetzstein, Maneesh Agrawala

We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. FramePack compresses input frame contexts with frame-wise importance so that more frames can be encoded within a fixed context length, with more important frames having longer contexts. The frame importance can be measured using time proximity, feature similarity, or hybrid metrics. The packing method allows for inference with thousands of frames and training with relatively large batch sizes. We also present drift prevention methods to address observation bias (error accumulation), including early-established endpoints, adjusted sampling orders, and discrete history representation. Ablation studies validate the effectiveness of the anti-drifting methods in both single-directional video streaming and bi-directional video generation. Finally, we show that existing video diffusion models can be finetuned with FramePack, and analyze the differences between different packing schedules.

Konrad Staniszewski, Szymon Tworkowski, Yu Zhao, Sebastian Jaszczur, H. Michalewski, Lukasz Kuci'nski, Piotr Milos

Recent advancements in long-context language modeling have attracted significant attention, yet their practical applications often suffer from suboptimal context utilization. To efficiently address this issue, we introduce the Structured Packing for Long Context, SPLiCe, a method that uses retrieval to collate mutually relevant documents into long training samples. We demonstrate that SPLiCe improves performance on long-context tasks, particularly by achieving perfect accuracy on the synthetic Needle in the Haystack benchmark, and effectively mitigating the ‘lost-in-the-middle’ phenomenon often observed in large language models. Notably, these long-context capabilities also extend to realistic downstream tasks, such as Qasper, across multiple model sizes—3B, 7B, and 13B—and are achieved with only brief fine-tuning on 2-6 billion tokens. We supplement these results with a detailed analysis of SPLiCe, examining the impact of hyperparameter choices, the different mixtures and proportions of SPLiCe-generated training data, and the choice of the retriever. We also study the transfer of long-context utilization skills between the modalities. An intriguing finding from our analysis is that training on a corpus of code can enhance performance on natural language tasks.

Saeed Kosari, Mirsaeid Hosseini Shirvani, Navid Khaledian, D. Javaheri

Anan Ashrabi Ananno, Luis Ribeiro

With the growing demand for sustainable and optimal packaging solutions, this study proposes a novel two-stage algorithm for the multi-container three-dimensional bin packing problem. The research addresses this problem within the context of a real-world industrial scenario and implements several practical constraints including: full shipment, customer positioning requirements, and product geometric interlocking, for increased stability and with the purpose of minimizing the use of plastic wrapping and/or additional supporting surfaces. The main optimization target is to minimize the total number of containers used in the palletization process of custom orders with varying degrees of complexity. The proposed algorithm includes two stages/phases of processing. In the first phase, the algorithm uses constructive heuristics to generate homogeneous product layers. The layers are then stacked to produce blocks, which are then placed on individual containers or pallets. The second phase packs the leftover items using a genetic algorithm. The performance of the proposed solution is benchmarked using real-world industrial data, as well as a more classic academic benchmark. It is demonstrated, across a very large set of orders, that the algorithm always achieves solutions for full palletization of the orders. The analysis shows that the approach is generic and the quality of the solutions generated is relatively even for both small and large, homogeneous and heterogeneous problem instances.