With the increasing implementation of machine learning models on edge or Internet-of-Things (IoT) devices, deploying advanced models on resource-constrained IoT devices remains challenging. Transformer models, a currently dominant neural architecture, have achieved great success in broad domains but their complexity hinders its deployment on IoT devices with limited computation capability and storage size. Although many model compression approaches have been explored, they often suffer from notorious performance degradation. To address this issue, we introduce a new method, namely Transformer Re-parameterization, to boost the performance of lightweight Transformer models. It consists of two processes: the High-Rank Factorization (HRF) process in the training stage and the deHigh-Rank Factorization (deHRF) process in the inference stage. In the former process, we insert an additional linear layer before the Feed-Forward Network (FFN) of the lightweight Transformer. It is supposed that the inserted HRF layers can enhance the model learning capability. In the later process, the auxiliary HRF layer will be merged together with the following FFN layer into one linear layer and thus recover the original structure of the lightweight model. To examine the effectiveness of the proposed method, we evaluate it on three widely used Transformer variants, i.e., ConvTransformer, Conformer, and SpeechFormer networks, in the application of speech emotion recognition on the IEMOCAP, M3ED and DAIC-WOZ datasets. Experimental results show that our proposed method consistently improves the performance of lightweight Transformers, even making them comparable to large models. The proposed re-parameterization approach enables advanced Transformer models to be deployed on resource-constrained IoT devices.

The rapid advancements in deep learning have enabled the development of models capable of addressing a wide range of tasks across domains such as natural language processing, computer vision, and time series forecasting (Vaswani et al., 2017; Devlin et al., 2018). However, predicting financial market behavior, especially identifying price surges in cryptocurrency markets, remains a challenging problem due to the stochastic nature of financial data and the influence of external factors (Benth et al., 2003; Cont, 2001). In recent years, Transformer-based models have demonstrated exceptional performance in time series forecasting by capturing long-range dependencies and temporal interactions(Vaswani et al., 2017; Lim and Zohren, 2021; Zhou et al., 2021). Simultaneously, the emergence of large language models (LLMs) has paved the way for transfer learning applications in financial time series data, including cryptocurrency markets (Raffel et al., 2020; Liu et al., 2019). This study introduces BreakGPT, an architecture that combines the strengths of LLMs and Transformer-based models for predicting cryptocurrency price surges. We evaluate multiple architectures, including a modified TimeLLM (Doe and Lee, 2023) and TimeGPT (Smith and Johnson, 2023), assessing their effectiveness in detecting price surges in assets like Bitcoin and Solana(Nakamoto, 2008; Zhang and McGovern, 2019). Key contributions of this study include: Development of a modified TimeLLM architecture that adapts GPT-2 for time series prediction using domain-specific prompts and embeddings (Doe and Lee, 2023; Radford et al., 2019). Implementation and comparison of various Transformer-based models that utilize attention mechanisms and convolutional layers to process financial time series data.

The results of the Anomalous Diffusion Challenge (AnDi Challenge) have shown that machine learning methods can outperform classical statistical methodology at the characterization of anomalous diffusion in both the inference of the anomalous diffusion exponent alpha associated with each trajectory (Task 1), and the determination of the underlying diffusive regime which produced such trajectories (Task 2). Furthermore, of the five teams that finished in the top three across both tasks of the AnDi challenge, three of those teams used recurrent neural networks (RNNs). While RNNs, like the long short-term memory (LSTM) network, are effective at learning long-term dependencies in sequential data, their key disadvantage is that they must be trained sequentially. In order to facilitate training with larger data sets, by training in parallel, we propose a new transformer based neural network architecture for the characterization of anomalous diffusion. Our new architecture, the Convolutional Transformer (ConvTransformer) uses a bi-layered convolutional neural network to extract features from our diffusive trajectories that can be thought of as being words in a sentence. These features are then fed to two transformer encoding blocks that perform either regression or classification. To our knowledge, this is the first time transformers have been used for characterizing anomalous diffusion. Moreover, this may be the first time that a transformer encoding block has been used with a convolutional neural network and without the need for a transformer decoding block or positional encoding. Apart from being able to train in parallel, we show that the ConvTransformer is able to outperform the previous state of the art at determining the underlying diffusive regime in short trajectories (length 10-50 steps), which are the most important for experimental researchers.