Toward Total-System Trustworthiness

Communications' Inside Risks columns have long stressed the importance of total-system awareness of riskful situations, some of which may be very difficult to identify in advance. Specifically, the desired properties of the total system should be specified as requirements. Those desired properties are called emergent properties, because they often cannot be derived solely from lower-layer component properties, and appear only with respect to the total system. Unfortunately, additional behavior of the total system may arise--which either defeats the ability to satisfy the desired properties, or demonstrates that the set of required properties was improperly specified. In this column, I consider some cases in which total-system analysis is of vital importance, but generally very difficult to achieve with adequate assurance.