Algorithm 1 demonstrates the step-by-step operations of our beam search algorithm (see Sec. 4.3). We consider recovering sentences in the current work. We leave recovering longer paragraphs as future work. We keep 2000 examples of each dataset as the evaluation set, and use the left for training. "End-to-End optimization", "Reg" means the inclusion of a regularization term, "DR" refers to a discrete token Our approach is unique as it does not rely on end-to-end optimization, is demonstrated on large batch sizes (i.e.

This paper introduces BSPA, a parallel algorithm that leverages beam search to address the two-dimensional strip packing problem. The study begins with a comprehensive review of existing approaches and methodologies, followed by a detailed presentation of the BSPA algorithm. Experimental results demonstrate the effectiveness of the proposed method. To facilitate further research, both the code and datasets are publicly available.

In the domain of artificial intelligence, two-player board games have historically served as pivotal'toy problems' for exploring and advancing search and planning algorithms within vast decision spaces. The outstanding algorithm AlphaZero (Silver et al. [2016] Silver et al. [2017a] Silver et al. [2017b]) achieved superhuman performance in the game of Go, chess, and other board games without the use of human expertise in these games. In this work, we introduce a new approach to solving such games. The main idea is that the algorithm iterates through possible moves using beam search, and then learns to predict the outcome of this search. This concept gives rise to the name of the algorithm, PROBS - Predict Results of Beam Search. This approach shows promising results -- it demonstrates an increase in the winning percentage during the training process and shows improvement with the use of greater computational power. Although this new approach to solving board games does not improve upon state-of-the-art approaches, it demonstrates a new working concept that may inspire researchers to develop new methods in other areas. The foundation of the PROBS algorithm is the iterative training of two neural networks. The first network is a value function, V (s), which predicts the expected utility from the current state.

The problem of decomposing an arbitrary Clifford element into a sequence of Clifford gates is known as Clifford synthesis. Drawing inspiration from similarities between this and the famous Rubik's Cube problem, we develop a machine learning approach for Clifford synthesis based on learning an approximation to the distance to the identity. This approach is probabilistic and computationally intensive. However, when a decomposition is successfully found, it often involves fewer gates than existing synthesis algorithms. Additionally, our approach is much more flexible than existing algorithms in that arbitrary gate sets, device topologies, and gate fidelities may incorporated, thus allowing for the approach to be tailored to a specific device.

Standard Recurrent Neural Network Transducers (RNN-T) decoding algorithms for speech recognition are iterating over the time axis, such that one time step is decoded before moving on to the next time step. Those algorithms result in a large number of calls to the joint network, which were shown in previous work to be an important factor that reduces decoding speed. We present a decoding beam search algorithm that batches the joint network calls across a segment of time steps, which results in 20%-96% decoding speedups consistently across all models and settings experimented with. In addition, aggregating emission probabilities over a segment may be seen as a better approximation to finding the most likely model output, causing our algorithm to improve oracle word error rate by up to 11% relative as the segment size increases, and to slightly improve general word error rate.

Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process. Although the programs they generate achieve high token-matching-based scores, they often fail to compile or generate incorrect outputs. The main reason is that conventional Transformer decoding algorithms may not be the best choice for code generation. In this work, we propose a novel Transformer decoding algorithm, Planning-Guided Transformer Decoding (PG-TD), that uses a planning algorithm to do lookahead search and guide the Transformer to generate better programs. Specifically, instead of simply optimizing the likelihood of the generated sequences, the Transformer makes use of a planner to generate candidate programs and test them on public test cases. The Transformer can therefore make more informed decisions and generate tokens that will eventually lead to higher-quality programs. We also design a mechanism that shares information between the Transformer and the planner to make our algorithm computationally efficient. We empirically evaluate our framework with several large language models as backbones on public coding challenge benchmarks, showing that 1) it can generate programs that consistently achieve higher performance compared with competing baseline methods; 2) it enables controllable code generation, such as concise codes and highly-commented codes by optimizing modified objective.

The transducer architecture is becoming increasingly popular in the field of speech recognition, because it is naturally streaming as well as high in accuracy. One of the drawbacks of transducer is that it is difficult to decode in a fast and parallel way due to an unconstrained number of symbols that can be emitted per time step. In this work, we introduce a constrained version of transducer loss to learn strictly monotonic alignments between the sequences; we also improve the standard greedy search and beam search algorithms by limiting the number of symbols that can be emitted per time step in transducer decoding, making it more efficient to decode in parallel with batches. Furthermore, we propose an finite state automaton-based (FSA) parallel beam search algorithm that can run with graphs on GPU efficiently. The experiment results show that we achieve slight word error rate (WER) improvement as well as significant speedup in decoding. Our work is open-sourced and publicly available\footnote{https://github.com/k2-fsa/icefall}.

Previous work on Chinese Knowledge Base Question Answering has been restricted due to the lack of complex Chinese semantic parsing dataset and the exponentially growth of searching space with the length of relation paths. This paper proposes an efficient pipeline method equipped with a pre-trained language model and a strategy to construct artificial training samples, which only needs small amount of data but performs well on open-domain complex Chinese Question Answering task. Besides, By adopting a Beam Search algorithm based on a language model marking scores for candidate query tuples, we decelerate the growing relation paths when generating multi-hop query paths. Finally, we evaluate our model on CCKS2019 Complex Question Answering via Knowledge Base task and achieves F1-score of 62.55\% on the test dataset. Moreover when training with only 10\% data, our model can still achieves F1-score of 58.54\%. The result shows the capability of our model to process KBQA task and the advantage in few-shot learning.

Pre-trained generative language models (such as OpenAI's GPT2 and GPT3) or seq2seq models (such as T5 or the recently released T0) generate free-flowing natural language. This entails that their output sentences can have any shape. To get the most value out of these models, we sometimes want the outputs to follow a certain structure -- this is called a constrained language model. After reading this post, you will be able to gain more value out of your language model by controlling the shape of its outputs. This post relates to my previous post, where I explain how to unlock the true potential of GPT3 by manipulating the shape of its inputs (referred to prompt engineering).

The research of adversarial attacks in the text domain attracts many interests in the last few years, and many methods with a high attack success rate have been proposed. However, these attack methods are inefficient as they require lots of queries for the victim model when crafting text adversarial examples. In this paper, a novel attack model is proposed, its attack success rate surpasses the benchmark attack methods, but more importantly, its attack efficiency is much higher than the benchmark attack methods. The novel method is empirically evaluated by attacking WordCNN, LSTM, BiLSTM, and BERT on four benchmark datasets. For instance, it achieves a 100\% attack success rate higher than the state-of-the-art method when attacking BERT and BiLSTM on IMDB, but the number of queries for the victim models only is 1/4 and 1/6.5 of the state-of-the-art method, respectively. Also, further experiments show the novel method has a good transferability on the generated adversarial examples.