Data Integration

Introducing Dagster - Nick Schrock - Medium


Today the team at Elementl is proud to announce an early release of Dagster, an open-source library for building systems like ETL processes and ML pipelines. We believe they are, in reality, a single class of software system. We call them data applications. Dagster is a library for building these data applications. We define a data application as a graph of functional computations that produce and consume data assets.

Extract, Shoehorn, and Load

Communications of the ACM

A lot of data is moved from system to system in an important and increasing part of the computing landscape. This is traditionally known as ETL (extract, transform, and load). While many systems are extremely good at this process, the source for the extraction and the destination for the load frequently have different representations for their data. It is common for this transformation to squeeze, truncate, or pad the data to make it fit into the target. This is really like using a shoehorn to fit into a shoe that is too small.

CloverDX Drinks and Data Meetup - London


Data integration software and ETL tools provided by the CloverDX platform (formerly known as CloverETL) offer solutions for data management tasks such as data integration, data migration, or data quality. CloverDX is a vital part of enterprise solutions such as data warehousing, business intelligence (BI) or master data management (MDM). CloverDX Designer (formerly known as CloverETL Designer) is a visual data transformation designer that helps define data flows and transformations in a quick, visual, and intuitive way. CloverDX Server (formerly known as CloverETL Server) is an enterprise ETL and data integration runtime environment. It offers a set of enterprise features such as automation, monitoring, user management, real-time ETL, data API services, clustering, or cloud data integration.

Data virtualization use cases cover more integration tasks


Gartner predicts that 60% of organizations will deploy data virtualization software as part of their data integration tool set by 2020. That's a big jump from the adoption rate of about 35% the consulting and market research company cited in a November 2018 report on the data virtualization market. But the technology "is rapidly gaining momentum," a group of four Gartner analysts wrote in the report. The analysts said data virtualization use cases are on the rise partly because IT teams are struggling to physically integrate a growing number of data silos, as relational database management system (DBMS) environments are augmented by big data systems and other new data sources. They also pointed to increased technology maturity that has removed deployment barriers for data virtualization users.

Predicting Crop Losses using Machine Learning CGIAR Platform for Big Data in Agriculture


Timely and accurate agricultural impact assessments for droughts are critical for designing appropriate interventions and policy. These assessments are often ad hoc, late, or spatially imprecise, with reporting at the zonal or regional level. This is problematic as we find substantial variability in losses at the village-level, which is missing when reporting at the zonal level. In this paper, we propose a new data fusion method--combining remotely sensed data with agricultural survey data--that might address these limitations. We apply the method to Ethiopia, which is regularly hit by droughts and is a substantial recipient of ad hoc imported food aid.

TTEK and Altana Trade Announce Alliance Agreement to Deliver Integrated Technology Solutions for Trade Risk Management Markets Insider


NEW YORK, May 27, 2019 /PRNewswire-PRWeb/ -- TTEK Inc. and Altana Trade today jointly announced their Alliance Agreement to deliver integrated technology solutions in trade risk management. As part of the alliance, TTEK will promote Altana Trade's products for data fusion and artificial intelligence-based risk management to government agencies worldwide. Altana will similarly incorporate TTEK's risk analytics and risk management workflow software in its solutions for financial services providers, multinationals, and logistics providers. TTEK is a rapidly growing provider of customs administration and risk management software and consulting services. With a global customer base and a team comprised of more than 150 of the world's preeminent customs modernization and risk management experts, TTEK helps to facilitate legal trade, enable systems interoperability and inter-agency coordination, and selectively target and inspect high-risk border flows.

How to Solve Data Integration Challenges with AI


Data is streaming through businesses at increasingly faster rates, creating time-to-insight and time-to-action data integration challenges which can be crucial to rising above competitors. As fast access to information becomes a greater demand for businesses, a more apparent challenge is in condensing data into useful information for developing insights. Many companies are spending more time and resources on increasing data and preparing it than they are on analysis. Agile, efficient, flexible data integration strategies are crucial to expanding big data analysis. If you're looking for easier ways to handle data integration projects, you may want to try looking into introducing artificial intelligence (AI) capabilities to your data integration strategies toolkit.

Noise-cancelling headsets worn by soldiers can reveal the position of a sniper after a single shot

Daily Mail - Science & tech

Locations of enemy snipers shooting at troops may soon be revealed instantly on the smartphones of the ambushed troops. Cutting-edge audio technology is being developed to use microphones in the ears of the soldiers to track two notable noises from a bullet - supersonic air in front of the bullet and the blast as it leaves the muzzle. Technology is being developed to use these two sounds to trace the original location and reveal where it was fired from. The data and location will then be relayed to the handset of the beleaguered troops to help them identify and neutralise the threat. Audio experts that developed the technology say it builds on existing technology and could be employed on the battlefield in just two years.

Data Integration Engineer - IoT BigData Jobs


C3 IoT has an opening for a Data Integration Engineer. You will be required to create advanced application integration solutions and configure, deploy and enhance enterprise cloud applications. C3 IoT product suite is entirely data-driven, so a great candidate will have a passion for acquiring, analyzing, and transforming data to generate insight. Qualified candidates will have a solid knowledge of integration and data manipulation technologies.

Technical Challenges and Lack of Data Science Talent Fettering Development of AI


The next move is to utilise notebooks which orchestrate both open source, proprietary AI intellectual property, and operational AI software code for specific domains. These notebooks allow scientists to follow recipes and best practices which often can't be learned in school or not available in organisations where the history of successful deployment of operationalised AI is limited or non-existent. To address the IT hurdles these notebooks increasingly are developed specifically with views to directly outputting operationalisation AI and scoring systems, for example utilising docker containers and Kubernetes to allow for ease of use of productionised components that are proven operational.