Mind

4.1 Mind as a physical thing

Our brains are also physical, apparently inert grey goo, yet full of electrochemical activity running through networked neurons and flooded with neurotransmitters. Three different kinds of physical process working together: electrical, structural (connectivity) and chemical.

In the 19th century, many believed that one could tell a person’s character and skills through the shape of the skull, reading ‘bumps’. This study of ‘phrenology‘ did make sense: if the brain is the seat of character and intellect, and if some parts of your brain are bigger then normal, then surely this should be able to tell you about the person. In fact, we now know that even gross measurements do not correlate well with specific abilities. For example, the total volume of the brain typically varies between individuals by up to 50 per cent, but there is no connection between this and intelligence.

However, while the raw size of parts of the brain is not related to specific traits or abilities, the physical parts of the brain do perform different jobs; there is functional separation. This was first discovered through accidents involving traumatic injuries to parts of the brain, or through tumours that affected specific parts of the brain. Nowadays the locations of these would be determined by X-rays or brain scans, but in the early days doctors often had to wait for the patients to die to study the brain during an autopsy.

Brain surgeons now use this knowledge to avoid damaging areas associated with critical functions such as Broca’s area and Wernicke’s area, which are essential for different aspects of language. However, the same surgery has also offered fresh knowledge. While the broad structures of the brain are roughly similar between individuals, the details vary, and in order to determine precisely the limits of critical areas surgeons will often use a probe to administer tiny electrical impulses to different parts of the exposed brain. The patients, still awake despite having part of their skull removed, perform simple tasks. If the impulse disrupts the task, this tells the surgeon which areas to avoid, and in the process reveals more about the fine structure of the brain. For example, stimulating an area may trigger a particular memory of an experience.

Modern brain scanning has allowed scientists to study the patterns of activity of the brain less invasively. In one example, which reveals the rich nature of imagination, subjects are placed in a brain scanner and then shown a picture. The scanner shows the parts of their brain that are active in response, including the visual system and the other parts connected with the interpretation of the picture. Then the picture is removed and the subjects are asked to imagine the picture. A few new parts light up, but there is much in common with the pattern when seeing the actual picture. In particular, parts of the visual cortex light up — the imagining of the picture stimulates parts of the sensory system in a similar way to actually seeing it.

BOX: fetal development of the brain, cortex as folded skin, the sea inside the children’s film Inside Out (Pixar) takes an interesting analogy of how the brain works, with memory & emotion

4.2 Memory and time

If we consider memory we see the importance of all three forms of brain physicality: electrical, connective and chemical [KV09]:

• short-term memory — Our fleeting thoughts are carried electrically, ionic discharges across synapses transferring information from neuron to neuron, as is measured through EEG monitoring. This is the basis of all our immediate perception, and also short-term memory with its 7+/-2 chunks of information [Mi56].
• long-term memory — In contrast, long-term memory is normally assumed to be stored in the physical connections between neurons and the strength of these connections. This storage literally grows. Even the strength is determined by the size of the synapse.
• mezzanine memory — There are things that lie between short-term and long-term memory: the general sense of where I am and what I am doing, and the memories of what I have been doing over recent minutes and hours that have not had time to ‘grow’ long-term memory and yet last longer than the effervescence of short-term electrical activity. This is less well studied in the psychological literature except for work in situation awareness [En95] and ‘long term working memory’ in studies of reading comprehension [EK95]. Indeed there is no name for this medium-term memory, and the name ‘mezzanine memory’ is Alan’s own. It seems likely that this form of memory is stored using chemical build-up with neurons, a process known as ‘long term potentiation’. [CB06;Lo03]

Chemicals in the form of neurotransmitters are also constantly produced in our brains, and hormones flood through our bodies. Some of these chemical processes are quite fast, in the order of seconds, but some, especially those in our bloodstream, may take many minutes to return to their normal state — think how long you feel ‘jumpy’ after a scare. Adrenaline and other chemicals flood through your body and do not disappear equally fast once you realise there is no danger.

While not part of our explicit memory systems, these chemicals act as a slowly decaying trace of past feelings and activity. In particular, many of these relatively slow-acting chemicals are critical to mood and emotion, and this has its impact on digital design.

Imagine visiting a friend who is about to have a party, then moving on to another who has lost a parent. We adapt and react differently to each, but in between our bodies have had time to ‘reset’ the chemicals that influence mood.

Now imagine the same scenario, but using instant messaging chat windows. On one side of the screen, planning a party; on the other, a funeral. While the electrical parts of our brains may be able to adapt rapidly between situations, the chemical parts cannot.

4.3 Just numbers

Because long-term memory is stored physically in the connections between neurons, it is possible to calculate just how much a single brain can remember [Dx05]. We have approximately 10 billion neurons and each neuron connects to between 1000 and 10,000 other neurons, giving around 100 trillion connections. It is this pattern of connections that is normally assumed to encode our memories. Each of these neurons also has a strength of connectivity given by the size of the synapse.

Now imagine a brain scanner in the far-off future that can trace each neuron, including how strong they are. As it reads the neurons it gives each one a number (needing about 34 bits) and records the connections and strength using a scale of 0 to 63. Together these require 5 bytes per connection. So with 100 trillion connections, our scanner of the future would need 500 trillion bytes, or half a petabyte, or approximately 500 hard disks, to store the neuron connectivity of a single human brain. Oddly, when the Internet Archive was set up in 1996, the dump of the web was about the same size [Dx05].

You can do a similar calculation for the speed of processing, to see how fast a computer would need to be to simulate the brain running in real time. The figure turns out to be about 10 petaflops, that is 10 thousand million million calculations per second. Sounds a lot, but yes, you guessed, if you add up the processing power of all the PCs connected to the Internet, it is about the same.

So in storage capacity and processing power the Internet is roughly similar to a single human brain!

However, although the raw numbers are roughly similar, the web is not really like a brain (and the brain is not like the web). Our brains are relatively slow (around 100 ‘cycles’ per second), but very highly parallel — lots of things happen at once. Each PC works much faster, several billion cycles per second, but serially, one thing at a time (although a 100 million PCs on the web is quite a lot of parallelism in itself!). Furthermore, the brain has a lot of non-local connections, that is neurons in one part of the brain talk to neurons in distant parts. This would make it very hard to actually emulate a brain on all those PCs, as the Internet in between would both be too slow and buckle under the load.

Does the idea of simulating a brain worry you? It is possible to simulate fluid flows, so one could model a glass of water, but running the computer simulation of the glass of water would not make you less thirsty any more than looking at a painting of a mountain stream.

The difference between simulating water and simulating our brains is that some would say that our thoughts, minds, consciousness, indeed who we are, reduces to the memory patterns and ‘execution’ of the brain functions. In other words, a computer emulating your brain is ‘just the same’ as you thinking it in your head.

Of course, these are things hotly contested in the philosophy of consciousness, with some arguing that the special nature of neurons makes a difference [Se97]. Roger Penrose even argues that it is the quantum effects of sub-cellular microtubules that allow us to think in ways that are not computable and therefore not able to emulated by a computer (except perhaps in a quantum computer) [Pe90].

When we discuss ‘mind and body’ we shall see a different view that places the human body as a crucial part of the picture, and some argue that no emulation of the mind could be truly a mind without being connected to a living, acting body.

4.4 Multiple intelligences

We will discuss the particular ways we think about objects and about space later in this book, but it is interesting to note that there are specific ways in which we think about objects that are not the same as the way we think about space, and different again from the way we think about other people. We pick up objects, move them around, and expect that they may be moved by others or natural forces. In contrast, with space, buildings, and the topography of hills and mountains, we are thinking about moving ourselves around and within them.

To some extent these can be seen as workings of the same sort of reasoning applied to different kinds of situation and indeed this is certainly the case for more logical thinking. However, there is strong evidence that we also have quite specialised ways of thinking about different aspects of the world, sometimes called ‘multiple intelligences‘ [Ga83;Ga93]. Some evidence for this comes from brain science, which shows there are specialised areas of the brain related to particular aspects of perceiving, thinking and acting (to the extent that these are separate!), and some evidence from experimental studies which show that effectively identical problems are solved differently if presented in ways which appeal to our different ‘intelligences’.

Imagine a pack of cards, each with a letter on one side and a number on the other. Four cards are selected and placed on a table like this:

You now need to verify whether the following rule is true of the four cards:

if there is a vowel on one side then there is an even number on the other

Which cards do you need to turn over to verify the rule?

This puzzle is called the Wason card test [Wa66] and variants of it have been used heavily in psychological experiments.

In fact, about three-quarters of people get the answer wrong. The most common answer given is the ‘E’ and ‘2’, but the ‘correct’ answer is actually the ‘E’ and ‘7’. The reason for the ‘incorrect’ answers is that we have a tendency to look for ‘confirming’ evidence, so we look at the back of the ‘2’, even though if it were not a vowel it would not invalidate the rule. We also focus on the cards most similar to concepts mentioned in the question, namely the vowel and the even number. In the real world, full of irrelevant information, these are sensible heuristics, but they do not work so well for artificial logic problems like this one.

Now consider a different problem. A newspaper stand has an ‘honesty box’ in which to put your money when you take a newspaper. A social psychologist watches people walking by. For each person the researcher writes on a small card. On one side she writes whether the person paid any money, on the other whether they took a newspaper. She even records a card for people who simply walk past.

Four of these cards are placed on the table:

You now need to verify whether the following rule is true of the four cards:

if the person takes a newspaper then they have paid money

Just as in the Wason card test, your task is to turn over as few cards as possible to verify the rule. Which cards do you turn over?

The problem is formally the same as the normal Wason card test, just substitute ‘newspaper’ for vowel, ‘money’ for even number. However, this time about three-quarters of people get the right answer. The explanation given for this is that this variant recruits knowledge about fairness and cheating and so is able to tap into our social intelligence, rather than abstract logic.

Some ‘evolutionary psychologists‘ talk about our brains being not so much a general-purpose tool (like a survivalist’s machete), but more like a Swiss Army Knife, with specialist tools to tackle special tasks [Co89;TC97].

In ‘The Prehistory of the Mind’ [Mi96] Steven Mithen argues that one of the crucial human developments has been the integration of these different intelligences, which in other animals, including early hominids, are distinct and unable to ‘communicate’ with one another. In other words the defining human cognitive accomplishment is joined-up thinking!

Mithen’s analysis is not merely speculative, but based on detailed analysis of the palaeontological record. Early hominids over a million years ago were able to shape flint tools, showing advanced forethought. Indeed even chimpanzees do this in the wild, selecting sticks of an appropriate size and shape, stripping them of leaves and breaking them to length so that they can later be used to dig out termites. Tool making itself is complex, but not uniquely human. However, it was not until 60,000 to 70,000 years ago that modern homo sapiens tied flint heads to wooden handles to make better axes or spears. This is seen as evidence that wooden sticks (previously living things) could now be thought about together with physical (non-living) stones.

Similarly it is only recently that we see the emergence of cave art and the making of clay or stone objects to represent animals or people. The representation of an animate thing (creature or human) using an inanimate medium (clay, stone or paint) again shows evidence of joined-up thinking.

To describe the cognitive architecture of the mind, Mithen uses the metaphor of a cathedral with (in order of evolutionary development) a nave of ‘general intelligence’, several chapels for the specialised intelligences, and a ‘super chapel’ of meta-representation where the differing intelligences can intermingle with each other leading to ‘cognitive fluidity‘. The form of this meta-representation is not elaborated although Mithen suggests it is in some way related to language, and this accords with the social changes occurring at around the same period, 50,000 to 60,000 BC, which are also often attributed to language development, a sort of socio-linguistic Eden.

Indeed the most obvious meeting point of different kinds of intelligence is in rational logical thought mediated by language. Words by their nature make things equal. While very different in meaning, ‘stone’, ‘spider’, ‘speed’, ‘society’, and ‘science’ are all ‘just’ words, tokens that can be put inside similar constructions. We can reason equally with all, and can mix them together, leading to a level of integration across intelligences 4.3.

However, as you may have guessed, with our focus on physicality that is not the end of the story.

Simpler creatures than ourselves, such as dogs, birds, and fish, do not lead fragmented existences, but are integrated because they live in the world. When we act based on a single kind of intelligence, the results of the actions are available to all of our modes of thinking, not just the one where they originated. If you want to go to see a colleague (social intelligence) and they are in the next room, you may start to walk towards them, but the fact that a wall is in the way will immediately be obvious to your spatial understanding (and if you do walk forward the actual presence of the wall will make it even more obvious). Things do not need to be linked in the head, because they are linked in the world (fig. 4.4).

However, while we do not always need to link forms of thinking in our heads, of course we often do, but apparently without lots of intense logical thinking. Partly this is due to imagination. As soon as you consider going to see your colleague next door an image of going starts to form. However, recall that imagination works its way right back into the sensory system, so it is almost as if you are actually seeing it, and this is available to your spatial intelligence just as with the real world. The physical intelligence says ‘no’ without you ever getting up from your chair. The ‘virtual world’ in your head kicks back just as hard as the real world (fig. 4.5).

4.5 The brain as interface

Most of our interactions with computers use our limbs, or maybe our voice, for input to control the computer, and sight, hearing or occasionally touch to get feedback from the computer. However, there are a number of technologies that more directly connect computer to brain, bypassing our voluntary body movements and our normal senses.

We have already seen one example of this, the implants into Kevin Warwick’s arm nerves that enabled him to control devices and directly connect to his wife. It is normal to use these nerves to control muscle movement and bodily feelings. So in a way this is bodily interaction — just bypassing the actual body!

With different technology one can even bypass the nerves and tap directly into the brain. This is often called ‘brain–computer interaction‘.

We have discussed how different parts of the brain perform different functions. Brain scanners can reveal this, but are a little large for use in computer interfaces. However, simpler EEG equipment can be used instead. EEG (electroencephalography) is used for medical diagnosis and involves placing large numbers of electrodes on the skull and measuring the electrical activity induced in the scalp by the corresponding activity within the brain due to the firing of neurons. Clinical apparatus may involve many dozens of electrodes, but in computer interaction research useful information can also be obtained using just a small number, often fitted into a skull-cap.

For many years EEG has been used in simple bio-feedback equipment, perhaps playing a different tone or projecting colours. In particular it is often used together with the detection of particular frequencies of activity (alpha and beta waves) associated with states of alertness or relaxation. While we do not normally have conscious control over these states of the brain, once they are made visible or audible in some way, subjects can often learn to influence them.

Bio-feedback is more about allowing the user to monitor their own body. However, if the outputs of EEG equipment are put into a machine learning algorithm, it is possible for the computer to recognise some of the user’s ‘thoughts’, though only from a small pre-chosen set. For example, there has been some success using apparatus of this kind to control an onscreen pointer by having subjects simply think about moving it one way or another. The recognition rate is still relatively poor, so this is not currently an option for day-to-day computing. For those patients with severe disabilities, particularly those with ‘locked-in syndrome’, both non-invasive use of EEG (or similar apparatus) and direct brain implants offer a chance to be able to communicate with the world.

There has also been some limited success in work with brain implants for visual impairment. Visual images are transferred from the retina to some parts of the visual cortex in a fairly direct way: there is something like (but not exactly like) a little image of the scene that you can see in front of you in the visual cortex. By working the other way round, brain implants that stimulate the visual cortex appear as points of light.

Less invasive means can be used for more everyday interactions, especially in affective gaming, that is games that respond in some way to the user’s emotional state. Most of these make use of heartbeat, or GSR (galvanic skin response — basically sweating as used in lie detectors). Both are related to the general level of arousal of the body, and so can be used as a rough measure of emotion. The most common form of game is ‘relax to win’, as you become more relaxed you get more speed or energy. This may be combined with social situations in which your friends try to make you less relaxed!

These physiological measures tell you how strong an emotion is, but not whether the person is very happy, or very angry, nor do they detect subtle emotions such as apprehension. Typically, more detailed emotion detection needs to use facial expression or tone of voice, both of which are more complex yet still only deliver a fairly small palette of emotions. Kiel Gilleade’s research has tried to bridge this gap by using the fact that as humans we are successful (when we are!) at detecting other people’s emotions, because we also know the context [GDA05]. In games one has a lot of information about the context, as it is a simulated world, and hence it is possible to make some inferences even from simple physiological measures.

The trouble with attempting to ascertain brain state from body state is that the body is affected by many things as well as what you are thinking and feeling. A researcher at Bristol University, Cliff Randell, designed a small camera that he wore connected to a heart-rate monitor. He reasoned that the important events one might want to record would be marked by periods of heightened emotion, so a camera was triggered to take a photograph when his heart rate peaked. The technology worked in the lab, but when he wore it elsewhere he found the photos were full of the tops of flights of stairs.

While the lacuna between pure mind and computation remains, it is perhaps not as wide as it once was. The means of contact are still very physical (silicon chips in the brain, wires and electrodes on the skull), but intimate and direct.

4.6 Creativity and physicality

Creativity is surely a defining cerebral activity. Artists wait upon the inspiration of their muse and the results are translated into words, images or sound. But of course it is not quite like that. Painters rarely sit for hours and then paint, they just paint. They seem to think with their brush. Similarly a composer will write their score sitting at a piano and a poet will jot down phrases. We will discuss this embodiment of thought more generally in the next section, but certainly there is some connection between creativity and physical manifestations.

The same is true of problem solving. Some problems can be solved analytically, by decomposing the elements of the problem and then going fairly mechanically through cases. However, others are not amenable to procedural solutions. Psychologists talk of ‘insight problem solving’, puzzles that seem intractable, but where there is a sudden ‘aha!’ moment and they become obvious. Look at the matches below. If you remove the seven matches from the top boxes, it is easy to leave just three squares. But can you remove five matches to leave exactly three squares?

You may try and ‘do it in your head’, but it is more likely that you will experiment and play with real matches. You might need some time away from the problem too, what the psychologists call ‘incubation‘, and when you return the problem may have become easy. Both time engaged in the physical form of the problem and time completely divorced from it seem to be necessary.

One such psychologist, Ronald Finke, studied this very physical ‘playing with materials’ stage in more creative design [Fi90]. In a series of experiments he studied simple problems where subjects were asked to perform simple creative tasks: creating things (e.g. means of transport) from shapes such as rods and circles. In some conditions the subjects were first given the shapes to play with for a period and then asked to come up with a design. In others they were simply presented with the shapes and immediately asked to come up with a design. He found that those who had time to play in a non-goal-directed way were able to produce more ‘creative’ designs.

This series of experiments by Finke and his colleagues was the basis of the ‘geneplore‘ model of creativity [FW96], which has two stages. The first is a ‘generation’ phase (corresponding to the play), which is not directed towards the goal, during which the subjects develop ‘pre-inventive structures’, patterns and relations of the materials to be used. This is followed by an ‘exploration’ stage (slightly confusingly named) when the pre-inventive structures are put together to deliver the creative solution. While there have been criticisms of both the model and the methods, the work underlined the benefits of the use of physical materials (although interestingly people could also perform purely mental ‘play’ as well).

In our own observations of group design activities we have also seen some profound effects of the choice of design materials on the results obtained. As part of a workshop on the topic of Physicality, attendees were divided into small teams and given a design brief: ‘a hand-held device for producing light’. Each team was given one of three kits of design materials to use, either (a) paper and pencils, (b) card and glue, or (c) modelling clay (plasticine). Participants were only supposed to use their own materials (although some cheated!). Beyond this they were not told how to use the materials, but in fact the materials implicitly suggested ways of use. For example, no team in the paper and pencil group chose to fold or mould the paper to make a model. In normal design any or all of these materials would be used according to the preferences of the designer at a particular moment. However, in this exercise, teams of participants were given just one kind of material to work with. Thus we were performing something similar to the ‘breaching experiments’ [Ga67] discussed in Chapter 1: a deliberate disruption of human activity in order to bring to light aspects that are tacit or taken for granted.  In this case we were disrupting the ability to choose appropriate materials.

Analysis of this design exercise and other work reveals the subtle ways physical materials influence design [RH10]. In particular, we saw differences in the way groups investigated depth versus breadth of ideas (paper and pencil sketches and lists generated more ideas than physical materials, but the latter were elaborated in greater detail). There were also subtle ways in which the materials both constrained and also suggested ideas (rolling of paper, deformability of clay), reflecting the importance of constraints, seen in the creativity and problem-solving literature [OM02].

Note we carefully use the word ‘influence’, as the materials do not fully determine outcomes. For example, one group given physical materials ‘cheated’ and used paper and pencil to sketch, reflecting Buxton’s focus on the importance, in early design, of the indeterminacy of the sketch [Bx07] over the more concrete prototype. And human ingenuity can conquer the obstinacy of the material world: one group made a cuddly teddy bear out of paper by tailoring the paper as one would a suit of clothes.

References