Sean’s Public Notebook

Interfaces can enable entirely new modes of thought

Good interfaces allow us to explore a problem space using means or metaphors or interaction patterns that can transform our understanding of a problem entirely. The GUI was invented as a tool for thought, after all, so this shouldn’t be entirely surprising.

Take, for example, the color histogram in a photo editing application, such as Photoshop. It allows for the visualization of color levels in a photo that would not be possible to fully grasp by simply looking at the photo. It allows for manipulation of color and understanding of color in a way that would be impossible to understand without this particular representation. In fact, by directly manipulating the representation, you can also reason about the photo in a way that would also otherwise be difficult or impossible. (Using an external external referrent in cognition is about more than offloading memory.)

Michael Nielsen writes in Thought as a Technology:

In a similar way to language, maps etc, a computer interface can be a cognitive technology. To master an interface requires internalizing the objects and operations in the interface; they become elements of cognition. A sufficiently imaginative interface designer can invent entirely new elements of cognition …

And …

… what makes an interface transformational is when it introduces new elements of cognition that enable new modes of thought. More concretely, such an interface makes it easy to have insights or make discoveries that were formerly difficult or impossible.

Increasingly, I believe that for such an interface to be transformational, it needs to support all possible representation modalities (visual, kinesthetic, symbolic, maybe even auditory!). Good interfaces should support all three of Bruner's modes of representation. Kinesthetic might even be the most important, because direct manipulation and rearrangement of a representation (often through epistemic actions) seems to draw on deep-seated reasoning abilities that we can’t access through other means. (Epistemic actions are taken to think, not to progress toward a goal.)

References

Nielsen, M. (2016) Thought as a Technology.

Kirsh, D. (2009) Interaction, External Representation and Sense Making.