Automatic evaluation is an integral aspect of dialogue system research. The traditional reference-based NLG metrics are generally found to be unsuitable for dialogue assessment. Consequently, recent studies have suggested various unique, reference-free neural metrics that better align with human evaluations. Notably among them, large language models (LLMs), particularly the instruction-tuned variants like ChatGPT, are shown to be promising substitutes for human judges. Yet, existing works on utilizing LLMs for automatic dialogue evaluation are limited in their scope in terms of the number of meta-evaluation datasets, mode of evaluation, coverage of LLMs, etc. Hence, it remains inconclusive how effective these LLMs are. To this end, we conduct a comprehensive study on the application of LLMs for automatic dialogue evaluation. Specifically, we analyze the multi-dimensional evaluation capability of 30 recently emerged LLMs at both turn and dialogue levels, using a comprehensive set of 12 meta-evaluation datasets. Additionally, we probe the robustness of the LLMs in handling various adversarial perturbations at both turn and dialogue levels. Finally, we explore how model-level and dimension-level ensembles impact the evaluation performance. All resources are available at https://github.com/e0397123/comp-analysis.

Using a vocabulary that is shared across languages is common practice in Multilingual Neural Machine Translation (MNMT). In addition to its simple design, shared tokens play an important role in positive knowledge transfer, assuming that shared tokens refer to similar meanings across languages. However, when word overlap is small, especially due to different writing systems, transfer is inhibited. In this paper, we define word-level information transfer pathways via word equivalence classes and rely on graph networks to fuse word embeddings across languages. Our experiments demonstrate the advantages of our approach: 1) embeddings of words with similar meanings are better aligned across languages, 2) our method achieves consistent BLEU improvements of up to 2.3 points for high- and low-resource MNMT, and 3) less than 1.0\% additional trainable parameters are required with a limited increase in computational costs, while inference time remains identical to the baseline. We release the codebase to the community.

While the hope of the US returning to the moon has been temporarily dashed, Astrobotic CEO John Thornton expressed high hopes for its future Griffin lunar lander missions. 'What a wild adventure we were just on,' Thornton said. 'Certainly not the outcome we were hoping for and certainly challenging right up front.' Like the Peregrine, these robotic lunar landers are expected to serve as a scout for the NASA's Artemis astronauts before they make their own moon landing in 2026. The CEO and trained mechanical engineer described'victory' after'victory' as his team scrambled to make the most of the scrapped Peregrine mission.

North Korea tested a nuclear-capable underwater attack drone designed to destroy naval vessels and ports, it said Friday. North Korea's military said it conducted the test in the country's eastern waters in response to naval drills by the U.S., South Korea and Japan which ended Wednesday. The underwater drone is among a broad range of weapon systems North Korean dictator Kim Jong Un continues to test and develop as he expands his arsenal of nuclear-capable weapons. "Our army's underwater nuke-based countering posture is being further rounded off and its various maritime and underwater responsive actions will continue to deter the hostile military maneuvers of the navies of the U.S. and its allies," North Korea's Defense Ministry said in a statement. It added: "We strongly denounce the U.S. and its followers for their reckless acts of seriously threatening the security of (North Korea) from the outset of the year and sternly warn them of the catastrophic consequences to be entailed by them."

Case law plays a crucial role in legal research, particularly in the context of human rights. Many international human rights conventions, such as the European Convention on Human Rights (ECHR), are considered'living instruments', which means that human rights should be interpreted in light of present-day conditions and in accordance with developments in international law [1]. Fundamental human rights, such as the right to respect for private and family life, home, and correspondence as enshrined in Article 8 of the ECHR, serve as broad normative standards that (may) evolve in response to societal changes and international consensus. For example, the meaning of'correspondence' has significantly changed with the internet and the progression of technology, and also what is considered'family life' [2] or a'home' is ever-developing [3]. Consequently, the interpretation and application of human rights undergo continuous development, requiring legal scholars and practitioners to rely heavily on the case law established by international courts, such as the European Court of Human Rights (ECtHR). However, the volume of case law is ever-increasing, which makes it challenging for legal scholars to discover relevant cases and gain a comprehensive understanding of this vast amount of information.

Deep Neural Networks (DNN) have emerged as an effective approach to tackling real-world problems. However, like human-written software, DNNs are susceptible to bugs and attacks. This has generated significant interests in developing effective and scalable DNN verification techniques and tools. In this paper, we present VeriStable, a novel extension of recently proposed DPLL-based constraint DNN verification approach. VeriStable leverages the insight that while neuron behavior may be non-linear across the entire DNN input space, at intermediate states computed during verification many neurons may be constrained to have linear behavior - these neurons are stable. Efficiently detecting stable neurons reduces combinatorial complexity without compromising the precision of abstractions. Moreover, the structure of clauses arising in DNN verification problems shares important characteristics with industrial SAT benchmarks. We adapt and incorporate multi-threading and restart optimizations targeting those characteristics to further optimize DPLL-based DNN verification. We evaluate the effectiveness of VeriStable across a range of challenging benchmarks including fully-connected feedforward networks (FNNs), convolutional neural networks (CNNs) and residual networks (ResNets) applied to the standard MNIST and CIFAR datasets. Preliminary results show that VeriStable is competitive and outperforms state-of-the-art DNN verification tools, including $\alpha$-$\beta$-CROWN and MN-BaB, the first and second performers of the VNN-COMP, respectively.

Open information extraction (OpenIE) aims to extract the schema-free triplets in the form of (\emph{subject}, \emph{predicate}, \emph{object}) from a given sentence. Compared with general information extraction (IE), OpenIE poses more challenges for the IE models, {especially when multiple complicated triplets exist in a sentence. To extract these complicated triplets more effectively, in this paper we propose a novel generative OpenIE model, namely \emph{DualOIE}, which achieves a dual task at the same time as extracting some triplets from the sentence, i.e., converting the triplets into the sentence.} Such dual task encourages the model to correctly recognize the structure of the given sentence and thus is helpful to extract all potential triplets from the sentence. Specifically, DualOIE extracts the triplets in two steps: 1) first extracting a sequence of all potential predicates, 2) then using the predicate sequence as a prompt to induce the generation of triplets. Our experiments on two benchmarks and our dataset constructed from Meituan demonstrate that DualOIE achieves the best performance among the state-of-the-art baselines. Furthermore, the online A/B test on Meituan platform shows that 0.93\% improvement of QV-CTR and 0.56\% improvement of UV-CTR have been obtained when the triplets extracted by DualOIE were leveraged in Meituan's search system.

Collecting relevant and high-quality data is integral to the development of effective Software Vulnerability (SV) prediction models. Most of the current SV datasets rely on SV-fixing commits to extract vulnerable functions and lines. However, none of these datasets have considered latent SVs existing between the introduction and fix of the collected SVs. There is also little known about the usefulness of these latent SVs for SV prediction. To bridge these gaps, we conduct a large-scale study on the latent vulnerable functions in two commonly used SV datasets and their utilization for function-level and line-level SV predictions. Leveraging the state-of-the-art SZZ algorithm, we identify more than 100k latent vulnerable functions in the studied datasets. We find that these latent functions can increase the number of SVs by 4x on average and correct up to 5k mislabeled functions, yet they have a noise level of around 6%. Despite the noise, we show that the state-of-the-art SV prediction model can significantly benefit from such latent SVs. The improvements are up to 24.5% in the performance (F1-Score) of function-level SV predictions and up to 67% in the effectiveness of localizing vulnerable lines. Overall, our study presents the first promising step toward the use of latent SVs to improve the quality of SV datasets and enhance the performance of SV prediction tasks.

Domain shift poses a significant challenge in Cross-Domain Facial Expression Recognition (CD-FER) due to the distribution variation across different domains. Current works mainly focus on learning domain-invariant features through global feature adaptation, while neglecting the transferability of local features. Additionally, these methods lack discriminative supervision during training on target datasets, resulting in deteriorated feature representation in target domain. To address these limitations, we propose an Adaptive Global-Local Representation Learning and Selection (AGLRLS) framework. The framework incorporates global-local adversarial adaptation and semantic-aware pseudo label generation to enhance the learning of domain-invariant and discriminative feature during training. Meanwhile, a global-local prediction consistency learning is introduced to improve classification results during inference. Specifically, the framework consists of separate global-local adversarial learning modules that learn domain-invariant global and local features independently. We also design a semantic-aware pseudo label generation module, which computes semantic labels based on global and local features. Moreover, a novel dynamic threshold strategy is employed to learn the optimal thresholds by leveraging independent prediction of global and local features, ensuring filtering out the unreliable pseudo labels while retaining reliable ones. These labels are utilized for model optimization through the adversarial learning process in an end-to-end manner. During inference, a global-local prediction consistency module is developed to automatically learn an optimal result from multiple predictions. We conduct comprehensive experiments and analysis based on a fair evaluation benchmark. The results demonstrate that the proposed framework outperforms the current competing methods by a substantial margin.

Named Entity Recognition (NER) is a field of computer science and natural language processing (NLP) that deals with the identification and classification of named items in unstructured text. The items in question belong to predefined semantic types such as persons, locations, and organizations [Grishman and Sundheim, 1996a]. NER is today a key component in areas including machine translation [Babych and Hartley, 2003], question-answering [Mollá et al., 2006], and information retrieval [Guo et al., 2009]. A number of NER systems have been developed, particularly for English, but also for other languages, including Chinese [Liu et al., 2022] and French [Mikheev et al., 1999]. Early NER systems used algorithms based on handcrafted rules, lexicons, and spelling features [Rau, 1991]. Systems were subsequently developed that used algorithms based on machine learning [Nadeau and Sekine, 2007], neural networks [Collobert, 2011], and transformers [Labusch et al., 2019a].