In-context Ranking (ICR) is an emerging paradigm for Information Retrieval (IR), which leverages contextual understanding of LLMs by directly incorporating the task description, candidate documents, and the query into the model's input prompt and tasking the LLM to identify relevant document(s). While it is effective, efficiency is a significant challenge in this paradigm, especially as the candidate list grows due to quadratic/super-linear scaling of attention operation with context length. To this end, this paper first identifies inherent and exploitable structures in the attention of LLMs finetuned for ICR: (1) inter-document block sparsity: attention is dense within each document block but sparse across different documents in the context; and (2) query-document block relevance: the attention scores from certain query tokens to a document block in middle layers strongly correlate with that document's actual relevance. Motivated by these observations, we introduce BlockRank (Blockwise In-context Ranking), a novel method that adapts the attention operation in an LLM by (a) architecturally enforcing the observed inter-document block sparsity, reducing attention complexity from quadratic to linear without loss in performance, and (b) optimizing query-document block relevance for true relevant documents during fine-tuning using an auxiliary contrastive training objective, improving retrieval in attention. Experiments on BEIR, MSMarco and NQ with Mistral-7B demonstrate that BlockRank Mistral matches or outperforms existing SOTA listwise rankers and controlled fine-tuned baseline while being significantly more efficient at inference (4.7x for 100 MSMarco documents in context) and scaling gracefully to long-context shortlists, around 500 documents in-context (approximately 100K context length) within a second, presenting a scalable and effective solution for ICR.

This story is part of an ongoing series in which we highlight graduates of Capital One's Machine Learning Engineering Training Program (MLETP), a 160-hour program that teaches software and data engineers the skills necessary to work in machine learning and AI. Pradeep picked up the value of continuous learning from his mother, who earned multiple master's degrees and a Ph.D. in education. So after becoming a software engineer at Capital One in 2017, he was quick to embed himself in our culture of growth and development. Pradeep followed his curiosity and began developing skills in machine learning, a form of artificial intelligence that can automatically predict outcomes. Capital One uses machine learning to create real-time and intelligent customer experiences that bring simplicity to banking.

Go Moment is the home of the smart concierge Ivy that is well known in the hospitality industry. Singh is also a product design expert and public speaker, and blogs at RajSinghLA.com. Rathinam is also well known and loved in the Seattle tech community and as a mentor to local startups. I caught up with the two of them, in the course of curating and hosting the Rethink CX webinar series sponsored by Freshworks. Like Paddy, I live in the Seattle, Washington area.)

Go Moment is the home of the smart concierge Ivy that is well known in the hospitality industry. Singh is also a product design expert and public speaker, and blogs at RajSinghLA.com. Rathinam is also well known and loved in the Seattle tech community and as a mentor to local startups. I caught up with the two of them, in the course of curating and hosting the Rethink CX webinar series sponsored by Freshworks. Like Paddy, I live in the Seattle, Washington area.)

"I may find three bugs, but the way I report information could change the decisions that business people have made, and I think business people will value those who help them make better decisions." On this episode of the Continuous Testing Live podcast, well-known software testing entrepreneur Pradeep Soundararajan shares his views on the opportunities that AI offers to testers who keep an open mind and a strong focus on user experience. Soundararajan also shares two questions that testers should always ask themselves when working to make informed decisions that impact both their customers and the business. Click here to listen to the podcast. Noel: So when I think of AI and machine learning in the software testing industry, I envision people using it to take care of things the same way we think about test automation: something we use for functional testing. But when I read a blog you wrote, titled, "AI-driven Functional Testing," it got into how we might also be able to use AI for exploratory testing, which I had not seen anyone write or talk about before. I'm really curious about the word "driven" in that title and wanted to get you to maybe expand a bit on how AI can be used to not just "aid" your functional test efforts, but really "drive" them.

Helping brands personalize communication with their users using #AI #Marketing.