Physicality of Things

 

7.1 Physics and naïve physics

We all learnt Newton’s laws of motion in school.

I.    Every objects stays still or moves at a uniform speed in a straight line unless acted on by an external force.

II.    The rate of change of momentum is equal to the force applied (F=ma)

III.    For every action there is an equal and opposite reaction.

But do you recall how confusing these were when you first heard them?  They may be ‘true’ but they don’t seem like ‘real life’. Two millennia earlier when Aristotle considered motion he asserted that the speed of an object, not the acceleration, was proportional to force. You push something harder, it goes faster. This accords with day-to-day life. If you are pushing a heavy box, when you push harder it moves faster and when you stop pushing it stops. But this contradicts Newton’s first and second laws, which suggest that the object should keep on moving at constant speed.

Of course, the two accounts can be reconciled. The box stops because of the frictional force that acts against its motion, and according to the third law there is an equal and opposite reaction, so that the friction also ‘pushes’ the floor, which itself pushes against the ground and ultimately the earth spins a little slower or faster (yes, really!). Happily for the future of the earth, the effects of pushing are negligible (the earth is a LOT bigger than any box you push) and moreover as you originally pushed it, your feet pushed against the earth in the opposite direction, neatly balancing the effects!

While the physics of Newton and the physics of everyday life are not actually at variance, they are different. To apply Newtonian physics to complex phenomena requires complex analysis, so we have rules for the world that are simpler to apply. These common-sense rules are often called naïve physics [Ha90;SC94] or folk physics.

Here we are most interested in what this means for human understanding of physical things, but researchers in artificial intelligence and robotics are also interested in naïve physics, since it often allows more efficient and robust automatic reasoning than attempting to use ‘proper’ Newtonian physics. With Newtonian physics, a robot wanting to move a box would have to know how heavy the box was, the characteristics of the surface of the floor and box (to work out the friction), and the exact power output of its own motors. If it encountered a tiny bump or imperfection in the surface it would have to start over again. With naïve physics it simply begins by pushing a little, and then pushes harder until the box moves.

We have naïve physics rules for objects and gravity, ‘what goes up must come down’; for space and containment, ‘if a room has only one entrance, what goes in must stay in or come out’; and for fluid substances, ‘the amount of liquid stays the same when you pour it’. Many of these rules are learned from a very early age; one of the stages noted by Piaget, the founder of developmental psychology, is when a child learns about conservation of number, volume and area.

more on Piaget

Even the basic start point is different. Newtonian physics tends to reduce objects to equivalent ‘point masses’, ignoring all their other attributes. In contrast, James Gibson, the ecological psychologist of perception and action, argued that a person’s immediate perception of objects is holistic; they are surfaces (the floor of a room, the outside of a ball) [Gi79]. The physicist breaks objects down into their smallest constituent parts and most basic properties to analyse, whereas our naïve physics deals with the ways these parts and properties normally interact as a whole. The word ‘normally’ here is crucial. When faced with very unusual or unnatural situations the rules of naïve physics tend to go awry: think of walking on ice, a smooth frictionless surface, or pictures of space walking. Crucially technology disrupts ‘normal’.

7.2 Rules of physical things

In contrast to Newton’s three laws there are many, many rules of naïve physics. However, here are three simple ‘rules’ of physical things. They were originally formulated to help explain to students why it is harder to design software than to engineer physical objects:

• directness of (or proportional) effort — Small effort produces small effects, large effort produces large effects. If you push a pebble a little, it moves a little; if you push it a lot, it moves a lot.
• locality of effect — The effects of actions occur where and when you act. If you push something and then it moves later you are surprised, and only a magician would try to move something without touching it.
• visibility of state — Physical objects have complex shape and texture, but this is largely static. The dynamic aspects of state are very simple: location, orientation, velocity and rate of angular rotation. However, as humans we are not very good at assessing even the last of these!

These rules do have exceptions. For example, if a rock is precariously balanced a small push might send it rolling down the hillside (breaking directness of effort). Or if you accidentally put a glass down on the edge of a table and turn round you may be surprised to hear it fall and shatter a few moments later. The interesting thing is even these very natural breakings of the rules cause us surprise or, like magnetism’s action at a distance, seem like magic.

All of these rules are systematically broken by human technology, and in particular digital technology. Consider a mobile phone:

• no directness of effort — Dial one digit wrong and you may ring someone in a different country, not just next door.
• no locality of effect — The whole purpose of a phone is to ring people up — spatial non-locality; the alarm you set at night rings in the morning — temporal non-locality; and text messages break both spatial and temporal locality!
• no visibility of state — The phone is full of hidden state, from the address book in the phone itself to the whole Internet (which while not ‘in’ the phone, can appear on the screen and therefore appears to be part of it).

As noted, it is not only digital technology that breaks these rules. Even the most basic technology often seems to give us supernatural power. For example, a simple saw means that a small amount of effort allows one to cut through a large piece of wood that would be impossible to split by hand (breaking directness of effort), and a bow and arrow allows action at a distance. Mechanical items like a car have complex invisible state (look under the bonnet) and a chemical plant is very like digital technology in terms of complexity (open a valve at one end of the plant and pressure goes up at a vessel at the other end).

It is interesting that in many areas of modern life where there is physical complexity, such as in the chemical plant, digital technology is being used to augment or substitute some of the unnatural activities. In a plane where cables once ran from cockpit to wing flaps, wires now carry signals to actuators. In the chemical plant not only are many valves operated electronically, but sensors allow one to see the impact across the plant.

The last example is particularly significant as the sensors and visualisations in the chemical plant control room allow visibility at a distance and reveal things inside the vessels (pressure, flows) that one would not see by eye. The technology, by extending our senses, ameliorates the disorienting effect of the broken rules of physicality. Things distant and hidden become close and visible, so that the virtual world of the control room is closer to ‘normal’ physical reality than the situation a few years previously where the impact of actions was unseen.

7.3 Continuity in time and space

Each of the above rules is connected with some form of continuity. In scientific terms, digital computation is naturally discrete and discontinuous whereas physics (above the quantum level) tends to be about continuous processes. Indeed there is a special area of computer science, ‘hybrid systems‘, which focuses on models that allow these two worlds to meet [HY09]. More radically, within HCI, ‘status–event analysis‘ tries to treat both kinds of phenomena on an equal footing [Dx91;DA96;DL07].

Continuity is often broken in magic worlds and in science fiction where portals and teleportation allow us to move across space, or in the case of the Tardis [Ta09] across time, without touching the points in between.

more: Continuity and learning, funny functions, different properties ‘stuff’ theorists, atomism, boundaries and identity

7.4 Conservation of number and preservation of form

In the story of the Frog Prince [GG23], the princess eventually kisses the frog and it turns into a handsome prince. Although this transmutation is odd, we accept it in the magic world. However, imagine if the story had three frogs, which, when the last was kissed, became a single prince. Somehow this is much more surprising. Conservation of number seems more primitive than conservation of form.

In fact, this difference is borne out by studies of very young babies. Newborns cannot focus clearly, but as soon as a baby is able to focus it is possible to obtain a measure of interest or surprise by recording how long the baby stares at something before moving their eyes.

One experiment involves having a barrier and putting objects one by one behind the barrier. In the baseline condition, the experimenter puts two mice behind the barrier and then opens the barrier and the baby sees two mice. In other conditions the experimenter puts two mice behind the barrier, but when it drops there are three mice, or just one mouse. In the conditions where the number revealed does not match the number put in, the baby will stare for longer, is more surprised: the tiniest babies can assess number (well, at least up to three!). Experiments with animals find a similar effect: many animals can assess number (though not count).

An alternative test involves putting, say, a truck behind the barrier, but when the barrier is removed, a duck is there. In such cases the babies show less surprise and need to be much older before they realise something is ‘wrong’. In other words conservation of number is more primitive than conservation of form [De97]. The Frog Prince’s transformation is not a surprise at all for a very young baby!  This is of course very sensible for a developing baby: a single object, such as a toy or a mother’s face, looks different from moment to moment as it moves, catches different light, or when the mother smiles. We have to learn over time which differences are simply differences of perspective and which really represent different things.

7.5 Emotion and nostalgia

E-cards have been around for more than 20 years and in 2008 were estimated to be growing by 200 percent per year, so much so that there were fears that traditional cards would become a thing of the past [Ad08]. Yet in the UK over a billion paper Christmas cards were purchased from bricks and mortar stores in 2018, and millions more from electronic sites offering print-on-demand customisable cards [GC18]. Most of us have mementos, either on display on a mantelpiece, or in a box in drawer: a ticket from a football match, earrings you got for your 18th birthday, love letters, or a milk jug that belonged to your mother. Physical things that you can touch and pick up, often things that were touched and held by someone else. They mean something because of what they are, who has given them, and the memories they hold.

Physical characteristics like weight and texture are critical in our emotional reaction to objects. If a device feels too light it may be perceived as fragile, whereas weight is often associated with value. So industrial designers have to worry as much about the feel of an object as its appearance. Car designers can create 3D CAD models, view them in virtual reality environments, see what they look like as the sun catches their curves, but still there is something else beyond the look, the speed and the accessories.  Just stroke a hand along the bonnet; does it feel right?

Theories of experience and emotion have different terms for this. McCarthy and Wright’s analysis of technology as experience includes a sensual thread of experience [MW04], the way the look and feel of an artefact or product creates thrill, excitement or even fear. Don Norman, writing about emotional design, refers to visceral design [No03], that immediate, in your body, reaction where “physical features — look, feel and sound – dominate.” And it works: users of the first iPod still talk about the feel of that scroll wheel [Bu04].

Not only is the feel of an object a strong part of its aesthetic appeal, but likewise the way we act on an object expresses our own emotions. When we are upset we may slam doors or clatter cutlery. And not only doors and cutlery, have you ever kicked your car or thumped the keys on your computer. Of course such displays of emotion can be unfortunate if we actually dent the car or break the computer, but they are an important part of expressing ourselves, and something we can use in design.

Certainly physical movements in Xbox interactions are used to control the gameplay, but they can also be expressive, perhaps swinging a virtual tennis racket between shots, or cornering over-sharply in a car race. However, just like the danger of breaking things in real life, if emotionally expressive actions overlap with those used to control the game the two may conflict. So, it can be beneficial to deliberately leave certain actions or gestures uninterpreted in order to provide space for expression.

Alternatively the user’s expressive actions can be deliberately sensed in order to use that emotion as part of the device’s behaviour. For example, researchers at Glasgow Caledonian University used the fact that gamepad buttons include pressure sensors. They hypothesised that the pressure with which a button is pressed will give an indication of the level of arousal, and indeed found that increasing the difficulty of the game increased average button pressure [SB03].

This form of implicit detection of mood was used in the design of the Key Table as part of the Equator ‘curious home’ [GB07]. This was a small table placed just inside the door where people would put their keys when coming in. The force with which they put them down was measured, from gentle placement to slamming them down. Behind the table was a picture frame with a small motor. If the keys were put down hard the picture frame would move slightly, almost as if they were slammed with enough force to shake the whole house. People were left to interpret as they liked the angle of the picture.

Stephan Wensveen, Kees Overbeeke and Tom Djajadiningrat went a step further in their design for an alarm clock [WO02;WO04]. Instead of a conventional means to set the time, twelve sliders were used. When all the sliders were in the middle the alarm would ring 12 hours from the time it was set. However, if you wanted eight hours sleep you had many options, you could put eight of the sliders in the middle, or put them all two-thirds of the way to the middle, or indeed use any pattern where the sliders ‘added up’ to eight. The mapping to the functionality of the alarm (the time to ring) only used some of the potential space of settings of the clock, leaving the rest free for emotional expression. Studies of how patterns of slider movement related to emotional state allowed the system to build a model of how the owner was feeling when going to sleep and this was used to adapt the alarm sound in the morning.

7.6 All our senses

While the fine arts tend to separate senses — oil painting for the visual, orchestras for the aural — the emotive and visceral reaction to physical objects is typically not confined to one attribute like appearance, sound, or feel, but is about the way they work together. An initial sense of the quality and robustness of a device loses its integrity if it is too light when you pick it up, or squashes under your fingers. Contrast the tinny clatter of cheap saucepans with the duller thud when you put down a cast iron casserole dish. Our senses have adapted to react to a range of facets of the physical world and, without realising it, we come to expect a close integration between them all.

Sound is our second sense and crucial for speech. It is also a part of almost every interaction with a physical object. The oddness of silent films is not so much the lack of speech, but the lack of everyday sounds, and the eeriness of fog scenes in movies is not just that you cannot see, but that sound too is dulled. Players of video games know that the sound is an essential part of the play, both the ambient soundtrack and the noises of things: the roar of an engine, the boom of an explosion, or the thwack of a ball hitting a tennis racket.

Mechanical devices make noises (the click of a switch, or purr of a motor), often also linked to vibrations, and while we usually strive to reduce how noisy things are, total silence can be unnerving. We ‘read’ the sonic environment without noticing that we do so. Think of an old jungle film, they wake in the night and feel that something is wrong, only slowly realising it is because of the silence, all the animals have fled. Total silence can also be dangerous, indeed some electric cars make an artificial noise to avoid accidents.

In general, digital devices tend to be more silent in terms of physical action. For example, many buttons on small devices use flexible underlays, and dials do not audibly click. The importance of hearing the sound of pressing a key is evidenced most strongly by the fact that many keyboards generate a simulated key-click. In fact, sound is one of the easiest senses to recreate digitally and has been used very effectively to make digital actions emulate physical ones. Think about the ‘crunch’ when you throw a file into the virtual desktop rubbish bin.

In the real world the sound an object makes is closely associated with its physical properties. Hard materials tend to make stronger and higher pitched sounds than soft materials: the sharp clang of a dropped spanner versus the dull thud of cricket ball and bat. Large and full vessels make deeper pitched sounds while small or empty vessels make high-pitched sounds. The sound also depends on the kind of action: tap a table gently and the sound is different than if you hammer it with your fist.

blowing on bottles

Some years ago, in the spirit of Gibson’s analysis of visual perception, Bill Gaver proposed an ecological approach to auditory sound perception, analysing the different sound patterns in neutral environments, whether from solid objects (bumps, bangs, scrapes), liquids (bubbles, flows, splashes), or gases (explosions, wind). Figure 7.4 shows his typology of different kinds of sounds [Gav93].

Gaver suggested that these natural relationships between sound and physical action and material could be used to generate better digital sounds. For example, if you discard a small file it should make a higher note than throwing away a large file, or during the download of a large document the progress bar could be accompanied by the sound of water filling a container with the sound becoming lower pitched as the download nears completion. The utility of this approach was verified by later experiments [My94], but the application of physical sounds in the interface is still minimal. For example, macOS does include a number of sounds such as a ‘crunch’ when files are thrown in the trash, but the sound does not depend on the size of the file.

Arkola

The feel of objects is obviously crucial when we cannot see or look at them directly: reaching out for a glass of water in the night or switching radio stations when driving a car. However, even when we can see the object and our hands, we still rely on tactile properties. In experiments with different prototype mobile phone keyboards, the most profound differences in behaviour were found when real keys were replaced by a flat membrane [GW10] (see the Equinox Case Study in Chapter 20). While the membrane still allowed tactile feedback as you pressed a button, you could not simply feel the buttons under your fingers as you tried to locate a key.

Recent digital technologies often replace the projecting buttons and knobs of older consumer electronics with sleek flat panels or touchscreens. Some touchscreen phones attempt to ameliorate this by using the vibrator motor to give little ‘kicks’ as your finger moves over onscreen buttons and keys. The effect is not the same as feeling a real button, but is remarkably effective, given the simplicity of the mechanism.

Another example comes from the iDrive found in some BMWs, first mentioned in Chapter 2. A single dial was to be used for many different menu functions, each of which could have different numbers of options. The iDrive uses a haptic feedback device, a small motor that gives slight resistance as you try to turn the knob.

These are both examples where a loss of tactile or haptic feedback is being replaced with digital emulation. However, tactile and haptic properties can be a significant design resource. By choosing surfaces with different characteristics (smoother, rougher, sticky), or controls with the right level of resistance, the user’s hand can be naturally guided to the most important controls, or learn the layout by touch alone. Furthermore, vibrotactile technology (arrays of tiny vibrating pins), which has been used for some years in electronic Braille displays, is beginning to become a feasible alternative for many devices, allowing the possibility of dynamically adjusting textures of surfaces.
more on vibrotactile technology  maybe technology box with diagrams of pins etc.

 

References