Background: What is a systemic black box in Artificial Intelligence (AI) systems?

In the context of AI systems, a black box refers to a technical device, system or object viewed in terms of its inputs and outputs, with limited or no knowledge of its internal workings and processes. Its implementation is “opaque” (black). It is a device, system or object that has “lost track” of the inputs that informed its decision-making structure to produce outputs – or more precisely was never keeping track. This is a part of what is described as the “black box problem” that are challenging on many levels, for example:

• difficulty in fixing the AI systems when problematic outcomes are produced.
• “trust” questions of system robustness for handling a variety of contexts/situations; and
• inaccuracies, missing experiences/perspectives, embedded unequal power relations and discriminatory features in the data-worlds reflections/amplifications that are used to train and build them and then serve to reproduce those harms and inequities into future contexts and decision-making AI processes.

Several value structures and belief systems underpin and permeate within AI development where black boxes can almost be seen as a type of social norm: 

• The belief that black boxes are a necessary evil in the development of sufficiently accurate AI models that is supported by the dichotomous reasoning that equates increased complexity of the AI model with accuracy versus more “simple – easy to interpret” transparent AI models as not-so accurate.
• The model-objects assumptions that turn individuals into passive recipients of the algorithmic outcomes that are built “for them” and are designed and implemented for convenience of use. A user provides an input to produce an instant output without any knowledge of how this industrial knowledge sausage is made. This is also a powerful gatekeeping mechanism for inclusion and exclusion practices.

There have been responses to concerns around the black box problem, some regulatory in nature but predominantly in the form of Explainable AI (XAI) or transparent AI practices. XAI may take the form of increased documentation of process but more commonly frames the problem of the black box as a technical one, which can be solved through a better technical solution. However, this does not always lead to addressing concerns about who is and is not included within these processes, or concerns for interpretability, access, or data-decision making/distributive justice.