Pitfalls in Processing Infinite-Length Sequences with Popular Approaches for Sequential Data

One of the enduring challenges for the Machine Learning community is developing models that can process and learn from very long data sequences. Transformer-based models and Recurrent Neural Networks (RNNs) have excelled in processing long sequences, yet face challenges in transitioning to processing infinite-length sequences online, a crucial step in mimicking human learning over continuous data streams. While Transformer models handle large context windows efficiently, they suffer from quadratic computational costs, motivating research into alternative attention mechanisms. Conversely, RNNs, particularly Deep State-Space Models (SSMs), have shown promise in long sequence tasks, outperforming Transformers in certain benchmarks. However, current approaches are limited to finite-length sequences, which are pre-buffered and randomly shuffled to cope with stochastic gradient descent. This paper addresses the fundamental gap in transitioning from offline-processing of a dataset of sequences to online-processing of possibly infinite-length sequences, a scenario often neglected in existing research. Empirical evidence is presented, demonstrating the performance and limits of existing models. We highlight the challenges and opportunities in learning from a continuous data stream, paving the way for future research in this area.