software testing


Combinatorial Testing for Deep Learning Systems

arXiv.org Artificial Intelligence

Deep learning (DL) has achieved remarkable progress over the past decade and been widely applied to many safety-critical applications. However, the robustness of DL systems recently receives great concerns, such as adversarial examples against computer vision systems, which could potentially result in severe consequences. Adopting testing techniques could help to evaluate the robustness of a DL system and therefore detect vulnerabilities at an early stage. The main challenge of testing such systems is that its runtime state space is too large: if we view each neuron as a runtime state for DL, then a DL system often contains massive states, rendering testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to reduce the testing space while obtaining relatively high defect detection abilities. In this paper, we perform an exploratory study of CT on DL systems. We adapt the concept in CT and propose a set of coverage criteria for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems. We further pose several open questions and interesting directions for combinatorial testing of DL systems.


5 key drivers for software testing in 2018: Driving the digital change

#artificialintelligence

Software testing has seen two major transitions in the past decade. The advent of agile heralded the shift from manual to automation and then in recent times, DevOps is driving the 2nd wave of change for testing teams with continuous integration and delivery. Although, in many ways, we are seeing some of the old practices coming back to life again in testing. Let us explore some major drivers that are becoming extremely important in 2018 and will remain in the future. The DevOps paradigm and continuous delivery have led to a digital shake-up for enterprise IT.


Artificial Intelligence (AI) in Risk Based Testing

#artificialintelligence

Analyzing large amounts of production data by actual users is a very good risk-based approach to testing highly complex systems based on what users actually do instead of all of the theoretical permutations of what a system may need to support. Is software testing capable of employing A.I. approaches, or will human software testers soon be bested by computers as well? This white paper highlights one specific approach that is driven by analyzing large amounts of production data that describes actual usage by real users through the data taken behind them. How A.I. can help the 5 basic sources of test coverage Different ways in which A.I. in testing can also be used Risk-Based Testing-Which test cases are most important and most likely to encounter failure, and how do you limit the count?


What are the advantages of Artificial Intelligence in Testing? - Maruti Techlabs

#artificialintelligence

Testing is a vital process that guarantees customer satisfaction within an application and helps in safeguarding against potential failures that may prove to be detrimental down the line. It is a planned process where the application is assessed and analyzed under certain conditions to understand the overall threshold and risks involved in its implementation. With software development life-cycles becoming more complicated by the day and delivery time spans reducing, testers need to impart feedback and evaluations instantly to the development teams. Given the breakneck pace of new software and product launches, there is no other choice than to test smarter and not harder in this day and age. Releases that happened once a month, now occur on a weekly basis and updates are factored in on almost every alternate day.


AI in software testing has arrived. Here's why robots rule.

#artificialintelligence

A world where robots do all the testing seems a long way off. For starters, demand isn't there, said Diego Lo Giudice, an analyst and vice president at Forrester Research. Lots of platform makers are adding AI to software testing, but Lo Giudice isn't getting many calls from clients about this. From what he sees, Sypolt's experience is typical. "Once they try it, they like it and are interested in it," he said.


6 levels of AI-based testing: Have no fear, QA pros TechBeacon

#artificialintelligence

Are you worried that artificial intelligence (AI) will soon replace all visual testing and QA teams? Autonomous testing tools are here to help, not replace you. Besides, testers and QA teams should be spending more time thinking about business value--that is, putting more effort into thinking about how to test and ensure quality--rather than performing automation testing just as a means to an end. Here are the six levels of testing so you can avoid getting caught flat-footed in the AI testing revolution. You write code that tests the application, and you're happy because you can run the same tests again and again on every release.


With $10M in funding, Mabl brings machine learning to software testing

#artificialintelligence

Mabl, a startup that's coming out of stealth today, uses machine learning to make functional testing for developers as easy as possible. Mabl users don't have to write extensive (and often brittle) tests by hand. Instead, they show the application the workflow they want to test and the service performs those tests -- and even automatically adapts to small user interface changes.



5 Ways AI Is Changing Software Testing - DZone AI

#artificialintelligence

Human software testing has yet to lose its value, but artificial intelligence (AI) is steadily advancing. AI is already touted for its advanced capabilities that may soon eliminate the need for manual testing in software development.


How to Test a Chatbot -- Part 1: Why is it so hard ?

#artificialintelligence

There are well estabilished software testing techniques and metrics, but what makes testing chatbots different? Most chatbots are built on top a learning cloud services, which by definition keeps changing its behaviour. NLU/NLP-Services (Natural language understanding and processing) like Dialogflow, Wit.ai or LUIS are subject to constant training and improvement. Having a non-deterministic component in the system under test will make software testing useless, as soon as you cannot tell the reason for a failed test case -- a defect in the chatbot software or an improvement in the cloud service. And even more important, the test itself can and will have an impact on the cloud services as well: presenting a cloud service with the same test cases over and over again will distort the cloud services assumption of "real-life interactions", giving the test cases higher priority than they should have.