Collaborating Authors

Major U.S. bank, a pioneer in the use of machine learning models, teams with Protiviti to improve its model validation framework


Following the financial crisis of 2007-2008, regulators issued specific guidance to help banks reduce the risk of financial losses or other adverse consequences stemming from decisions based on incorrect or misused financial models. Since then, the guidance has become the model risk management bible for financial institutions. It is used to ensure that model validation, typically performed annually, can identify vulnerabilities in the models and manage them effectively. Recently, the rapid advance and broader adoption of machine learning (ML) models have added more complexity and time to the model validation process. Specifically, ML models have highlighted expertise gaps in in-house model validation teams trained in traditional modeling techniques.

Robustness, Adaptivity, and Resiliency Analysis

AAAI Conferences

In order to better understand the mechanisms that lead to resiliency in natural systems, to support decisions that lead to greater resiliency in systems we effect, and to create models that will utilized in highly resilient systems, methods for resiliency analysis will be required. Existing methods and technology for robustness analysis provide a foundation for a rigorous approach to resiliency analysis, but extensions are necessary to address the multiple time scales that must be modeled to understand highly adaptive systems. Further, if resiliency modeling is to be effective, it must be contextualized, requiring that the supporting software will need to mirror the systems being modeling by being pace layered and adaptive.

How To Fine Tune Your Machine Learning Models To Improve Forecasting Accuracy?


Fine tuning machine learning predictive model is a crucial step to improve accuracy of the forecasted results. In the recent past, I have written a number of articles that explain how machine learning works and how to enrich and decompose the feature set to improve accuracy of your machine learning models. I am often asked a question on the techniques that can be utilised to tune the forecasting models when the features are stable and the feature set is decomposed. Once everything is tried, we should look to tune our machine learning models. This diagram illustrates how parameters can be dependent on one another.

An Algorithmic Framework for Computing Validation Performance Bounds by Using Suboptimal Models Machine Learning

Practical model building processes are often time-consuming because many different models must be trained and validated. In this paper, we introduce a novel algorithm that can be used for computing the lower and the upper bounds of model validation errors without actually training the model itself. A key idea behind our algorithm is using a side information available from a suboptimal model. If a reasonably good suboptimal model is available, our algorithm can compute lower and upper bounds of many useful quantities for making inferences on the unknown target model. We demonstrate the advantage of our algorithm in the context of model selection for regularized learning problems.

The VIVA Method: A Life-Cycle Independent Approach to KBS Validation

AAAI Conferences

Steven A. Wells AIE Department, Lloyd's Register 29, Wellesley Road, Croydon CRO 2A J, England Abstract This paper describes the VIVA method; a lifecycle independent method for the validation of Knowledge-Based Systems (KBS). The method based upon the VIVA validation framework, a set of products by which a KBS development can be described. By assessing properties of these products, and properties of the links between the products, a framework for validation throughout the KBS life-cycle is defined. Introduction The VIVA 1 Technical Annex (VIVA 92) identifies the needs for a Knowledge-Based System validation method which covers the whole of the development lifecycle. These needs arise from identified problems with software-based validation, which can be summarised as follows: It is not possible to determine the validity of a system from the software alone.