Artificial Intelligence Impact Assessments ("AIIAs"), a family of tools that provide structured processes to imagine the possible impacts of a proposed AI system, have become an increasingly popular proposal to govern AI systems. Recent efforts from government or private-sector organizations have proposed many diverse instantiations of AIIAs, which take a variety of forms ranging from open-ended questionnaires to graded score-cards. However, to date that has been limited evaluation of existing AIIA instruments. We conduct a classroom study (N = 38) at a large research-intensive university (R1) in an elective course focused on the societal and ethical implications of AI. We assign students to different organizational roles (for example, an ML scientist or product manager) and ask participant teams to complete one of three existing AI impact assessments for one of two imagined generative AI systems. In our thematic analysis of participants' responses to pre- and post-activity questionnaires, we find preliminary evidence that impact assessments can influence participants' perceptions of the potential risks of generative AI systems, and the level of responsibility held by AI experts in addressing potential harm. We also discover a consistent set of limitations shared by several existing AIIA instruments, which we group into concerns about their format and content, as well as the feasibility and effectiveness of the activity in foreseeing and mitigating potential harms. Drawing on the findings of this study, we provide recommendations for future work on developing and validating AIIAs.

You might think that's something like Kubeflow but Kubeflow is more of a pipelining and orchestration system that's not really agnostic to the languages and frameworks that run on it.

We're already talking with several advanced data science engineering teams that are working on amazing open source projects that form the glue between different platforms, and we're looking to roll them under the Alliance." Of course, with so many moving parts to coordinate, fostering these emerging links hasn't been without challenges, and the AIIA is looking to learn from the missteps of similar precedents so that they can avoid making the same mistakes. "We've got to make sure that everyone sees the bigger picture and works together -- a rising tide lifts all boats," said Jeffries. "We don't want this to turn into a meaningless reference architecture. We don't want everyone in the Alliance pushing and pulling so hard that it warps the stack all out of proportion or collapses to individual interests. The trick here is to focus on mutual benefits -- every member of the Alliance must ask themselves how the Canonical Stack can help the Alliance as a whole. We also don't want governance by pure committee.