Differentiable Search Index (DSI) utilizes Pre-trained Language Models (PLMs) for efficient document retrieval without relying on external indexes. However, DSIs need full re-training to handle updates in dynamic corpora, causing significant computational inefficiencies. We introduce PromptDSI, a rehearsal-free, prompt-based approach for instance-wise incremental learning in document retrieval. PromptDSI attaches prompts to the frozen PLM's encoder of DSI, leveraging its powerful representation to efficiently index new corpora while maintaining a balance between stability and plasticity. We eliminate the initial forward pass of prompt-based continual learning methods that doubles training and inference time. Moreover, we propose a topic-aware prompt pool that employs neural topic embeddings as fixed keys. This strategy ensures diverse and effective prompt usage, addressing the challenge of parameter underutilization caused by the collapse of the query-key matching mechanism. Our empirical evaluations demonstrate that PromptDSI matches IncDSI in managing forgetting while significantly enhancing recall by over 4% on new corpora.

Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al.,2022)--from 125M to 175B parameters--on next-token prediction, sequence-level generation, and downstream tasks. We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent behavior; 2) early in training, all models learn to reduce the perplexity of grammatical sequences that contain hallucinations, with small models halting at this suboptimal distribution and larger ones eventually learning to assign these sequences lower probabilities; 3) perplexity is a strong predictor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independent of the model size. Together, these results show that perplexity is more predictive of model behaviors than model size or training computation.

To further explore the intersection of 5G and manufacturing, register for the 5G Manufacturing Forum. Global manufactuers are starting to adopt 5G to improve manufacturing processes. Low latency and high reliability are needed to support critical applications in the manufacturing field. Several top manufacturers are already taking advantage of 5G implementation to improve operations in different industrial environments. Here we briefly describe some implementations by large manufacturers globally.

Southfield, MI, June 2016 – Stefanini, a 1B global IT provider, announced today that the company is launching Sophie, its artificial intelligence platform with the ability to turn data into valuable solutions. Aware of the latest trends, Stefanini has invested and developed this platform over the last 3 years as a Research & Development and pilot project for clients in Brazil, and now, the company is launching the platform in the United States. "We are very proud to introduce Sophie for our clients in North America, reinforcing Stefanini's commitment to connect people and technology innovations with a goal to create business value," said Antonio Moreira, Stefanini CEO, North America and Asia Pacific. "Our artificial intelligence platform can improve the end-user experience and deliver smarter and more efficient services," affirmed Mr. Moreira. Technology research firm Gartner forecasts that by 2017, autonomics-based managed services and cognitive platforms will fuel a significant reduction in the cost of IT services by automating repetitive tasks currently tackled by humans.