Poor data quality limits the advantageous power of Machine Learning (ML) and weakens high-performing ML software systems. Nowadays, data are more prone to the risk of poor quality due to their increasing volume and complexity. Therefore, tedious and time-consuming work goes into data preparation and improvement before moving further in the ML pipeline. To address this challenge, we propose an intelligent data-centric evaluation framework that can identify high-quality data and improve the performance of an ML system. The proposed framework combines the curation of quality measurements and unsupervised learning to distinguish high- and low-quality data. The framework is designed to integrate flexible and general-purpose methods so that it is deployed in various domains and applications. To validate the outcomes of the designed framework, we implemented it in a real-world use case from the field of analytical chemistry, where it is tested on three datasets of anti-sense oligonucleotides. A domain expert is consulted to identify the relevant quality measurements and evaluate the outcomes of the framework. The results show that the quality-centric data evaluation framework identifies the characteristics of high-quality data that guide the conduct of efficient laboratory experiments and consequently improve the performance of the ML system.

In natural language processing (NLP) we always rely on human judgement as the golden quality evaluation method. However, there has been an ongoing debate on how to better evaluate inter-rater reliability (IRR) levels for certain evaluation tasks, such as translation quality evaluation (TQE), especially when the data samples (observations) are very scarce. In this work, we first introduce the study on how to estimate the confidence interval for the measurement value when only one data (evaluation) point is available. Then, this leads to our example with two human-generated observational scores, for which, we introduce ``Student's \textit{t}-Distribution'' method and explain how to use it to measure the IRR score using only these two data points, as well as the confidence intervals (CIs) of the quality evaluation. We give quantitative analysis on how the evaluation confidence can be greatly improved by introducing more observations, even if only one extra observation. We encourage researchers to report their IRR scores in all possible means, e.g. using Student's \textit{t}-Distribution method whenever possible; thus making the NLP evaluation more meaningful, transparent, and trustworthy. This \textit{t}-Distribution method can be also used outside of NLP fields to measure IRR level for trustworthy evaluation of experimental investigations, whenever the observational data is scarce. Keywords: Inter-Rater Reliability (IRR); Scarce Observations; Confidence Intervals (CIs); Natural Language Processing (NLP); Translation Quality Evaluation (TQE); Student's \textit{t}-Distribution

Get the Spectroscopy and Chemometrics News Weekly in real time on Twitter @ CalibModel and follow us. "Rapid prediction of Yongchuan Xiuya tea quality by using near infrared spectroscopy coupled with chemometric methods" LINK "An improved method for predicting soluble solids content in apples by heterogeneous transfer learning and near-infrared spectroscopy" LINK "Research on construction method and validity mechanism of robust analysis model in snow peach quality detection based on visible-near infrared spectroscopy" LINK "Identification of milk powder brands by visible-near infrared spectroscopy based on principal component analysis and neural networks" LINK "Near Infrared Spectroscopy coupled to Chemometrics for the authentication of donkey milk" LINK "Applied Sciences: Construction and Application of Detection Model for Leucine and Tyrosine Content in Golden Tartary Buckwheat Based on Near Infrared Spectroscopy" LINK "Fast and robust NIRS-based characterization of raw organic waste: using non-linear methods to handle water effects" LINK "Hazelnut quality detection based on deep learning and near-infrared spectroscopy" LINK "Soil Nitrogen Content Detection Based on Near-Infrared Spectroscopy" LINK "Rapid nondestructive detecting of sorghum varieties based on hyperspectral imaging and convolutional neural network" LINK "Detection of Water Content in Lettuce Canopies Based on Hyperspectral Imaging Technology under Outdoor Conditions" LINK "Near-Infrared Spectroscopy Coupled with Chemometrics and Artificial Neural Network Modeling for Prediction of Emulsion Droplet Diameters" LINK "How can cry acoustics associate newborns' distress levels with neurophysiological and behavioral signals?" "How can cry acoustics associate newborns' distress levels with neurophysiological and behavioral signals?" "Agriculture: The Application of Machine Learning Models Based on Leaf Spectral Reflectance for Estimating the Nitrogen Nutrient Index in Maize" LINK "Foods: A Method for Capture and Detection of Crop Airborne Disease Spores Based on Microfluidic Chips and Micro Raman Spectroscopy" LINK "Plants: Pattern Recognition of Varieties of Peach Fruit and Pulp from Their Volatile Components and Metabolic Profile Using HS-SPME-GC/MS Combined with Multivariable Statistical Analysis" LINK "Compositional analysis in sorghum (Sorghum bicolor) NIR spectral techniques based on mean spectra from single seeds" LINK "Nondestructive Techniques for Fresh Produce Quality Analysis: An Overview" LINK "Application of Spectroscopy for Assessing Quality and Safety of Fresh Horticultural Produce" LINK

Roads are critically important infrastructure to societal and economic development, with huge investments made by governments every year. However, methods for monitoring those investments tend to be time-consuming, laborious, and expensive, placing them out of reach for many developing regions. In this work, we develop a model for monitoring the quality of road infrastructure using satellite imagery. For this task, we harness two trends: the increasing availability of high-resolution, often-updated satellite imagery, and the enormous improvement in speed and accuracy of convolutional neural network-based methods for performing computer vision tasks. We employ a unique dataset of road quality information on 7000km of roads in Kenya combined with 50cm resolution satellite imagery. We create models for a binary classification task as well as a comprehensive 5-category classification task, with accuracy scores of 88 and 73 percent respectively. We also provide evidence of the robustness of our methods with challenging held-out scenarios, though we note some improvement is still required for confident analysis of a never before seen road. We believe these results are well-positioned to have substantial impact on a broad set of transport applications.

Maintaining data reliability is a resource-intensive, uphill task for many organizations. Companies often spend too much effort on data reviews and cleanup, but seldom seem to catch up. Most of the time, teams don't even know what the issues are, how to look for them and how to solve them. They just know that the data is dirty, and like sitting on a ticking time bomb, we wait for the disaster to happen. The issues are often only illuminated when the data is put to operational use and trips up the end user or the customer with wrong information.

Maintaining data reliability is a resource-intensive uphill task for many organizations. Companies often spend too much effort on data reviews and cleanup, but seldom seem to catch up. Most of the time, teams don't even know what the issues are, how to look for them, and how to solve them. They just know that the data is dirty, and like a sitting on a ticking time bomb, we wait for the disaster to happen. The issues are often only illuminated when the data is put to operational use and trips up the end user or the customer with wrong information.

Research over the past years has shown that machine translation results can be greatly enhanced with the help of mono- or bilingual human contributors, e.g. by asking hu- mans to proofread or correct outputs of machine translation systems. However, it remains difficult to determine the quality of individual revisions. This paper proposes a meth- od to determine the quality of individual contributions by analyzing task-independent data. Examples of such data are completion time, number of keystrokes, etc. An initial evaluation showed promising F-measure values larger than 0.8 for support vector machine and decision tree based classifications of a combined test set of Vietnamese and German translations.