GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

Specifically, GUI-AIMA-3B is better than strong large size coordinate-based UI-T ARS-1.5-7B, JEDI-7B, and also better than the embedding-based coordinate-free GUI-Actor-7B model, highlighting the superiority of directly supervising on the multi-head self-attention weights instead of modeling the query-visual attention map via hidden states. On ScreenSpot-v2, GUI-AIMA-3B achieve the comparable results with the strongest baselines, such as JEDI-7B and UI-T ARS-7B, and better than the same size GUI-Actor-3B, while GUI-AIMA-3B is trained with much less web data. Besides the advanced performance, another advantage is the training efficiency of GUI-AIMA with only 259k training elements, as most supervised fine-tuned baselines trained on millions of GUI elements. For the comparison with reinforcement fine-tuned baselines, while both trained with smaller training sets than SFT baselines, GUI-AIMA-3B performs better and shows better generalization. Among GUI-AIMA-3B and GUI-AIMA-3B (soft), GUI-AIMA-3B is slightly better on dealing with diverse graphic environments in ScreenSpot-v2, OSWorld-G and ScreenSpot-pro. The two-step inference with zoom-in without extra training significantly improves the performance of GUI-AIMA on high-resolution benchmarks, ScreenSpot-pro and OSWorld-G, specifically 59.6% on ScreenSpot-pro using GUI-AIMA-3B (soft) and 63.8% on OSWorld-G using GUI-AIMA-3B, demonstrating the flexibility of GUI-AIMA's attention-based patch-wise grounding for inference-time improvements.