- chapter 7 Physicality of Things
- section 7.1 physics and naïve physics
- box 7.1 naïve physics: weight and worth
- section 7.2 rules of physical things
- section 7.3 continuity in time and space
- section 7.4 conservation of number and preservation of form
- figure 7.1 Jessie M. King: The Frog Prince
- section 7.5 emotion and nostalgia
- figure 7.2 The emotionally expressive Key Table [SB03].
- figure 7.3 Emotionally expressive alarm clock [WO02;WO04]
- box 7.2 SenToy – playing with emotions
- section 7.6 all our senses
I. Every objects stays still or moves at a uniform speed in a straight line unless acted on by an external force.
II. The rate of change of momentum is equal to the force applied (F=ma)
III. For every action there is an equal and opposite reaction.
But do you recall how confusing these were when you first heard them? They may be ‘true’ but they don’t seem like ‘real life’. Two millennia earlier Aritstotle considered motion and he asserted that the speed of an object, not the acceleration was proportional to force. You push something harder, it goes faster. This accords with day-to-day life. If you are pushing a heavy box, when you push harder it moves faster and when you stop pushing it stops. But this contradicts Newton’s first and second laws, which suggest that the object should keep on moving at constant speed.
Of course, the two accounts can be reconciled. The box stops because of the frictional force that acts against its motion. And according to the third law there is an equal and opposite reaction, so that the friction also ‘pushes’ the floor, which itself pushes against the ground and ultimately the earth spins a little slower or faster (yes really!). Happily for the future of the Earth, the effects of pushing are negligible (the earth is a LOT bigger than any box you push) and moreover as you originally pushed it, your feet push against earth in the opposite direction, neatly balancing the effects!
Whilst the physics of Newton and the physics of everyday life are not actually at variance, they are different. To apply Newtonian physics to complex phenomena needs complex analysis, so the rules we have for the world are ones which are simpler to apply. These common sense rules are often called naïve physics [Ha90;SC94] or folk physics.
Here we are most interested in what this means for human understanding of physical things, but those working in artificial intelligence and robotics are also interested in this naïve physics as it often allows more efficient and more robust automatic reasoning than attempting to use ‘proper’ Newtonian physics. With Newtonian physics a robot wanting to move a box would have to know how heavy the box was, the characteristics of the surface of the floor and box (to work out the friction), the exact power output of its own motors, and if it encountered a tiny bump or imperfection in the surface would start over again. With naïve physics it simply starts by pushing a little, and then pushes harder until the box moves.
We have naïve physics rules for objects and gravity “what goes up must come down”, for space and containment “if a room has only one entrance, what goes in must stay in or come out”, and for fluid substances “the amount of liquid stays the same when you pour it”.
Even this naïve physics is not so ‘obvious’ that it doesn’t need to be learnt. One of the stages noted by Piaget, the founder of developmental psychology, is when a child learns about conservation of number, volume and area.more on Piaget
Even the basic start point, is not the same. Physics tends to reduce objects to equivalent ‘point masses’, where larger objects are composite. In contrast, Gibson, the ecological psychologist of perception, argued that surfaces (the floor, the outside of an object) and mediums (air) are the basic units of perception and action [Gi79].
The physicist breaks things down into their smallest constituent parts and most basic properties to analyse; whereas our naïve physics deals with the ways these normally work together on wholes. Of course the ‘normally’ here is crucial, when faced with very unusual or unnatural situations they tend to go awry: think of walking on ice, a smooth frictionless surface, or pictures of space walking. Crucially technology disrupts ‘normal’.
Box 7.1 naïve physics: weight and worth
A good general rule of thumb is that things of similar substance weigh more when they are bigger. A large sack of potatoes weighs more than a small one. However, if you are moving house the boxes all look similar, but contain very different things hidden inside. You may find yourself almost falling over when you try to pick up a large box that is nearly empty as your muscles prepared for the heavy weight. If you actually weigh boxes, you find that when you have a larger box that is exactly the same weight as a smaller one, the smaller one feels heavier than the large one. Because you expect the large one to be heavier your head is saying “this is lighter than I thought”.
When you pick up a small metal item, it similarly feels weightier compared to a larger plastic one that is actually identical in weight, and because it feels weightier physically, it also often feels weightier in terms of importance or quality.
In contrast to Newton’s three laws there are many many rules of naïve physics. However, here are three simple ‘rules’ of physical things. They were originally formulated to help explain to students why it is harder to design software compared to the engineering of physical things:
- directness of (or proportional) effort – Small effort produces small effects, large effort produces large effects. If you push a pebble a little, it moves a little; if you push it a lot, it moves a lot – simple Newtonian dynamics.
- locality of effect – The effects of actions occur where and when you act. If you push something and then it moves later you are surprised, and only a magician would try to move something without touching it.
- visibility of state – Physical objects have complex shape and texture, but this is largely static. The dynamic aspects of state are very simple: location, orientation, velocity and rate of angular rotation. However, as humans we are not very good at assessing even the last of these!
These rules do have exceptions. For example, if a rock is precariously balanced a small push might send it rolling down the hillside (breaking directness of effort). Or if you accidentally put a glass down on the edge of a table and turn round you may be surprised to hear it fall and shatter a few moments later. The interesting thing is even these very natural breakings of the rules cause us surprise, or like magnetism’s action at a distance, seem like magic.
All of these rules are systematically broken by human technology, and in particular digital technology, consider a mobile phone:
- no directness of effort – Dial one digit wrong and you may ring someone in a different country, not just next door.
- no locality of effect – The whole purpose of a phone is to ring people up – spatial non-locality; the alarm you set at night and then rings in the morning – temporal non-locality; and text messages break both spatial and temporal locality!
- no visibility of state – The phone is full of hidden state, from the address book in the phone itself to the whole internet (which not ‘in’ the phone, but can appear on the screen and therefore appears to be part of it).
As noted it is not just digital technology that breaks these rules, the power of even the most basic technology is often that it seems to give us super-natural power. For example, a simple saw means that a small amount of effort allows one to cut through a large piece of wood that would be impossible to break by hand (breaking directness of effort), and a bow and arrow allows action at a distance. Mechanical items like a car of course have very complex invisible state (look under the bonnet) and a chemical plant is very like digital technology in terms of complexity (open a valve one end of the plant and pressure goes up at a vessel at the other end).
It is interesting that in many areas of modern life where there is physical complexity, such as in the chemical plant, digital technology is being used to augment or substitute some of the unnatural activities. In a plane where cables once ran from cockpit to wing flaps, now wires carry signals to actuators. In the chemical plant not only are many valves operated electronically, but also sensors allow one to see the impact across the plant.
The last example is interesting as the sensors and visualisations in the chemical plant control plant allow visibility at a distance and reveal things inside vessels (pressure, flows) that one would not see by eye. The technology, by extending our senses, ameliorates the disorienting effect of the broken rules of physicality. In a way things distant and hidden become close and visible, so that the virtual world of the control room is closer to ‘normal’ physical reality than the situation a few years previously where the impact of actions was hidden or distant.
Most of the above rules are connected with some form of continuity. In scientific terms, digital computation is naturally discrete and discontinuous whereas physics (above the quantum level) tends to be about continuous processes. Indeed there is a special area of computer science ‘Hybrid Systems’ focused on models that allow these two worlds to meet [HY09] and ‘Status–Event’ analysis that tries to treat both kinds of phenomena on an equal footing [Dx91;DA96;DL07].
Continuity is often broken in magic worlds and in science fiction where portals and teleportation allows us to move across space, or in the case of the Tardis [Ta09] across time, without touching the points in between.more: Continuity and learning, funny functions, different properties ‘stuff’ theorists, atomism, boundaries and identity
In the story of the Frog Prince [GG23], the princess eventually kisses the frog and it turns into a handsome prince. Although this transmutation is odd, we accept it in the magic world. However, imagine if the story had three frogs, which, when the last was kissed, became a single prince. Somehow this is more shocking. Conservation of number seems more primitive than conservation of form.
Fig. 7.1 Jessie M. King: The Frog Prince
In fact, this difference is borne out by studies of very young babies. Newborns cannot focus clearly, but as soon as a baby is able to focus it is possible to obtain a measure of interest or surprise by recording how long the baby stares at something before moving their eyes.
One experiment involves having a barrier and putting objects one by one behind the barrier. In the baseline condition, the experimenter puts two mice behind the barrier and then opens the barrier and the baby sees two mice. In other conditions the experimenter puts two mice behind the barrier, but when it drops there maybe three mice or just one mouse. In the conditions where the number revealed does not match the number put in, the baby will stare for longer, is more surprised … the tiniest babies can assess number (well at least up to 3!). In addition experiments with animals find a similar effect, that is many animals can assess number (not count).
An alternative test involves putting, say, a truck behind the barrier, but then when the barrier is removed, a duck is there. In such cases the babies show less surprise and need to be much older before they realise something is ‘wrong’. In other words conservation of number is more primitive than conservation of form [De97]. The frog prince’s transformation is not a surprise at all for a very young baby! This is of course very sensible for a developing baby: a single object, such as a stick or a mother’s face looks different from moment to moment as it moves, catches different light, or when the mother smiles. We have to learn over time which differences are simply ones of perspective and which really represent different things.
Ecards have been around for more than 20 years and estimated to be growing by 200% per year, so much so that there have been fears that traditional cards will become a thing of the past. Yet still in the UK 750 million paper cards were sent in 2008 and in the US the figure is about triple that [Ad08;UC05;UC08]. Either on display on a mantelpiece, or in a box in drawer, most of us have mementos, a ticket from a football match, earrings you got for your 18th Birthday, love letters, or a milk jug that belonged to your mother. Physical things that you can touch, pick up and often that were touched and held by someone else. They mean something because of what they are, who has given them, and the memories they hold.
Physical characteristics, weight, texture are critical to our emotional reaction to it. If a device feels too light it may be perceived as fragile whereas weight is often associated with value. So industrial designers have to worry as much about the feel of an object as its appearance. Car designers can create 3D CAD models, view them in virtual reality environments, see what they look like as the sun catches their curves, but still there is something else beyond the look and speed and the accessories. Just stroke a hand along the bonnet; does it feel right?
Theories of experience and emotion have different terms for this. McCarthy and Wright’s analysis of technology as experience, includes a sensual thread of experience [MW04], the way the look and feel of an artefact or product creates thrill, excitement or even fear. Similarly, Don Norman writing about emotional design refers to visceral design [No03], that immediate in your body reaction where “physical features – look, feel and sound – dominate.” And it works: users of the first iPod still talk about the feel of that scroll wheel [Bu04].
Not only is the feel of an object a strong part of its aesthetic appeal, but likewise the way we act on an object expresses our own emotions. When we are upset we may slam doors or clatter cutlery. And not only doors and cutlery, have you ever kicked your car or thumped the keys on your computer. Of course such displays of emotion can be unfortunate if we actually dent the car or break the computer, but they can be an important part of expressing ourselves, and something we can use in design.
Certainly physical movement in Xbox interactions are used to control the gameplay, but they can also be expressive, maybe swinging a virtual tennis racket between shots, or cornering over-sharply in a car race. However, just like the danger of breaking things in real life, if emotionally expressive actions overlap with those used to control the game the two may conflict. So, it can be beneficial to deliberately leave certain actions or gestures uninterpreted in order to leave space for expression.
Alternatively the users expressive actions can be deliberately sensed in order to use that emotion as part of the device’s behaviour. For example, researchers at Glasgow Caledonian University used the fact that gamepad buttons include pressure sensors. They hypothesised that the pressure with which a button is pressed will give an indication of the level of arousal, and indeed found that increasing the difficulty of the game increased average button pressure [SB03].
This form of implicit detection of mood was used in the design of the Key Table as part of the Equator ‘curious home’ [GB07]. This was a small table placed just inside the door where people would put their keys when coming in. The force with which they put them down was measured, from gentle placement to slamming them down. Behind the table was a small picture frame with a small motor. If the keys were put down hard the picture frame would move slightly almost as if they were slammed with enough force to shake the whole house. People were left to interpret as they liked, the angle of the picture.
Fig. 7.2 The emotionally expressive Key Table [SB03].
Stephan Wensveen, Kees Overbeeke and Tom Djajadiningrat went a step further in their design for an alarm clock [WO02;WO04]. Instead of a conventional means to set the time twelve sliders were used. When all the sliders were in the middle the alarm would ring 12 hours from the time it was set; however if you wanted eight hours sleep you had many options, you could put 8 of the sliders in the middle, or put them all 2/3 of the way to the middle or indeed any pattern where the sliders ‘added up’ to eight. That is the mapping to the functionality of the alarm (the time to ring) only used some of the potential space of settings of the clock, leaving the rest free for emotional expression. Studies of how patterns of slider movement related to emotional state allowed the system to build a model of how the owner was feeling when going to sleep and this was used to adapt the alarm sound in the morning.
Box 7.2 SenToy – playing with emotions
Researchers in the European Safira project produced a small stuffed toy called SenToy [PC03]. SenToy is a bit like a rag doll except it has no features, simply limbs and head. However inside SenToy is a box of electronics including accelerometers to sense shaking and movement and cables for measuring flexing of the arms. SenToy is used to control a game of magic, however, unlike a normal video game you cannot directly influence your characters actions, but instead you control its emotional state; for example, if you stretch SenToy’s arms wide, your character is surprised, if you shake it violently it becomes angry. Inside the game a programme then creates appropriate behaviour dependent on the emotion.
After playing with SenToy, people do not want to give it back. Some of this is to do with the fact that the whole experience is about emotion, but that is not the whole story. A careful design exercise compared potential designs and found that the soft cuddly designs were chosen over hard plastic. The feel is as important as the behaviour.
While the fine arts tend to separate senses: oil painting for the visual, orchestras for the aural, the emotive and visceral reaction to physical objects is typically not confined to one attribute: appearance, sound, or feel; but is about the way they work together. The initial sense of quality and robustness of device loses its integrity if it is too light when you pick it up, or squashes under your fingers. Contrast the tinny clatter of cheap saucepans with the duller thud when you put down a cast iron casserole dish. Our senses have adapted to react to a range of facets of the physical world and, without realising it, we come to expect the close integration between them all.
Sound is our second sense and crucial for speech, but also a part of every interaction with a physical object. The oddness of silent films, is not so much the lack of speech, but the lack of everyday sounds, and the eeriness of fog scenes in movies is not just that you cannot see, but that sound too is dulled. Players of video games know that the sound is essential part of the play, both the ambient soundtrack, but also the noises of things: the roar of an engine, the boom of an explosion, or the thwack of a ball hitting a tennis racket.
Box 7.3 Silent Battles
Think of the space fights in a film like Star Wars. There is a constant sound of gunfire and explosions. However, in a real space fight you would only hear your own engines and gunfire. Enemy kills and the destruction of your own side would be silent except perhaps for the occasional clunk of debris against your hull. The only time you would hear an explosion would be your last.
Mechanical devices make noises (the click of a switch, or purr of a motor), often also linked to vibrations, and while we usually strive to reduce the noise of things, total silence can be unnerving. We ‘read’ the sonic environment without noticing that we do so. Think of an old jungle film, they wake in the night and feel that something is wrong, but only slowly realise it is because of the silence, all the animals have fled. Total silence can also be dangerous, indeed some electric cars make an artificial noise to avoid accidents.
In general, digital devices tend to be more silent in terms of physical action, for example many buttons on small devices use flexible underlays and dials do not audibly click. The importance of hearing the sound of pressing a key is evidenced most strongly by the fact that many keyboards generate a simulated key-click.
In fact sound is one of the easiest senses to recreate digitally and has been used very effectively to make digital actions emulate physical ones. Think about the ‘crunch’ when you throw a file into the virtual desktop rubbish bin.
In the real world the sound an object makes is closely associated with its physical properties. Hard materials tend to make stronger and higher pitched sounds than soft materials: the sharp clang of a dropped spanner vs. the dull thud of cricket ball and bat. Large and full vessels make deeper pitched sounds whilst small or empty vessels make high-pitched sounds. The sound also depends on the kind of action: tap a table gently and the sound is different if you hammer it with your fist.blowing on bottles
Some years ago, in the spirit of Gibson’s anaysis of visual perception, Bill Gaver proposed an ecological approach to auditory sound perception analysing the different sound patterns in neutral environments, whether from solid objects (bumps, bangs, scrapes), liquids (bubbles, flows, splashes), or gasses (explosions, wind); figure 7.4 shows his typology of different kinds of sounds [Gav93].
Fig. 7.4 Typology of ecological sounds from [Gav93].
Gaver suggested that these natural relationships between sound and physical action and material could be used in order to generate better digital sounds; for example, if you discard a small file it should make a higher note than throwing away a large file, or during the download of a large document, the progress bar could be accompanied by the sound of water filling a container with the sound becoming lower pitched as the download nears completion. The utility of this approach was verified by later experiments [My94], however, the application of physical sounds in the interface is still minimal. For example, MacOS does include a number of sounds such as a ‘crunch’ when files are thrown in the trash, but the sound does not depend on the size of the file.Arkola
The feel of objects is obviously crucial when we cannot see or look at them directly: reaching out for a glass of water in the night or switching radio stations when driving car. However, even when we can see the object and our hands, we still rely on tactile properties. In experiments with different prototype mobile phone keyboards, the most profound differences in behaviour were found when real keys were replaced by a flat membrane [GW10] (see the Equinox Case Study in Chapter 20). While the membrane still allowed tactile feedback as you pressed a button, you could not simply feel the buttons under your fingers as you try to locate a key.
Recent digital technologies often replace the projecting buttons and knobs of older consumer electronics with sleek flat panels or touch screens. Some touch-screen phones attempt to ameliorate this by using the vibrator motor to give little ‘kicks’ as your finger moves over on-screen buttons and keys. The effect is not the same as feeling a real button, but is remarkably effective given the simplicity of the mechanism.
The aforementioned iDrive found in some BMWs is another example. A single dial was to be used for many different menu functions, each of which could have different numbers of options. The iDrive uses a haptic feedback device, a small motor that gives slight resistance as you try to turn the knob.
These are both examples where a loss of tactile or haptic feedback is being replaced with digital emulation. However, tactile and haptic properties can be a significant design resource. By choosing surfaces with different characteristics (smoother, rougher, sticky) or controls with the right level of resistance, the user’s hand can be naturally guided to the most important controls, or learn the layout by touch alone. Furthermore vibrotactile technology (arrays of tiny vibrating pins), which has been used for some years in electronic Braille displays, is beginning to become a feasible alternative for many devices, allowing the possibility of dynamically adjusting textures of surfaces.
more on vibrotactile technology maybe technology box with diagrams of pins etc.
1. [Ad08] Stephen Adams. Traditional Christmas card ‘set for decline’. The Telegraph. 15 Dec 2008. http://www.telegraph.co.uk/scienceandtechnology/technology/3776875/Traditional-Christmas-card-set-for-decline.html [7.5]
2. [Bu04] Eliot Van Buskirk. Perspective: The secret of iPod’s scroll wheel. CNET. September 22, 2004. http://news.cnet.com/2010-1041_3-5375101.html [7.5]
3. [De97] Dehaene, S. (1997). The Number Sense. Penguin, London. [7.4]
4. [Dx91] Dix, A.: Status and events: static and dynamic properties of interactive systems, Proc. of the Eurographics Seminar: Formal Methods in Computer Graphics, 1991. http://www.hcibook.com/alan/papers/euro91/ [7.3]
5. [DA96] Dix, A. and Abowd, G.: Modelling Status and Event Behaviour of Interactive Systems, Software Engineering Journal, 11:6, 334–346 (1996). http://www.hcibook.com/alan/papers/SEJ96‐s+e/ [7.3]
6. [DL07] Dix, A., Leite, J. and Friday, A.: XSED – XML‐based Description of Status–Event Components and Systems, Proc. of Engineering Interactive Systems 2007 (EIS 2007), LNCS 4940, Springer, 2008. http://www.hcibook.com/alan/papers/EIS‐DSVIS‐XSED‐2007/ [7.3]
8. [GB07] Gaver, W., Bowers, J., Boucher, A., Law, A., and Pennington, S. (2007). Electronic furniture for the Curious Home: Assessing ludic designs in the field. International Journal of Human-Computer Interaction, 22(1-2): 119-152. [7.5]
9. [Gi79] Gibson, J. (1979). The Ecological Approach to Visual Perception. New Jersey, USA, Lawrence Erlbaum Associates [7.1]
10. [GW10] Gill, S., Walker, D., Loudon, G., Dix, A., Woolley, A., Ramduny-Ellis, D. and Hare, J. (2010). Rapid Development of Tangible Interactive Appliances: Achieving the Fidelity/Time Balance. In Hornecker, E., Schmidt, A., and Ullmer, B. (eds) Special Issue on Tangible and Embedded Interaction, International Journal of Arts and Technology, Volume 1, No 3/4 pp 309-331. [7.6]
11. [GG23] Grimm,, Jacob W. and Wilhelm K. Grimm. The Frog Prince. Grimm’s Fairy Tales, Edgar Taylor and Marian Edwardes (Translators), 1823. Current editions: Norilana Books, 2008, Penguin Books 2007. Online text at Project Guttenberg: http://www.gutenberg.org/files/2591/2591‐h/2591‐h.htm#2H_4_0013 [7.4]
12. [Ha90] Hayes, P. J. 1990. The second naive physics manifesto. In Readings in Qualitative Reasoning About Physical Systems, D. S. Weld and J. d. Kleer, Eds. Morgan Kaufmann Publishers, San Francisco, CA, 46-63. [7.1]
13. [HY09] HYSCOM , IEEE Control System Society (CSS) has a Technical Committee on Hybrid Systems, accessed March 2009. http://www.dii.unisi.it/hybrid/ieee/ [7.3]
14. [My94] Elizabeth D. Mynatt. 1994. Designing with auditory icons: how well do we identify auditory cues?. In Conference Companion on Human Factors in Computing Systems (CHI ’94), Catherine Plaisant (Ed.). ACM, New York, NY, USA, 269-270. DOI=http://dx.doi.org/10.1145/259963.260483 [7.6]
15. [No03] Donald A. Norman, Emotional Design: why we love (or hate) everyday things, New York: Basic Books, 2003, pp.257, ISBN 0465051359. [7.5]
16. [PC03] Ana Paiva, Marco Costa, Ricardo Chaves, Moises Piedade, Dario Mourao, Daniel Sobral, Kristina Hook, Gerd Andersson, Adrian Bullock, SenToy: an affective sympathetic interface, International Journal of Human-Computer Studies, Volume 59, Issues 1-2, Applications of Affective Computing in Human-Computer Interaction, July 2003, Pages 227-235, ISSN 1071-5819, DOI: 10.1016/S1071-5819(03)00048-X. http://dx.doi.org/10.1016/S1071-5819(03)00048-X [7.5]
17. [SC94] Barry Smith and Roberto Casati. Naive Physics: An Essay in Ontology. Philosophical Psychology, 7/2 (1994), 225-244. [7.1]
18. [MW04] John McCarthy and Peter Wright, Technology as Experience, The MIT Press, 2004. ISBN:0262134470 [7.5]
19. [SB03] Sykes, J. and Brown, S. 2003. Affective gaming: measuring emotion through the gamepad. In CHI ’03 Extended Abstracts on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA, April 05 – 10, 2003). CHI ’03. ACM, New York, NY, 732-733. DOI= http://doi.acm.org/10.1145/765891.765957 [7.5;7.5]
20. [Ta09] TARDIS, Wikipedia page. http://en.wikipedia.org/wiki/TARDIS [7.3]
21. [UC05] US Census Bureau. The Holiday Season. Facts for Figures. CB05-FF.19-2 December 19, 2005. http://www.census.gov/Press-Release/www/releases/archives/facts_for_features_special_editions/005870.html [7.5]
22. [UC08] US Census Bureau. The Holiday 2008 Season. Facts for Figures. # CB08-FF.21. Oct. 30, 2008. http://www.census.gov/Press-http://www.census.gov/Press-Release/www/releases/archives/facts_for_features_special_editions/012876.html [7.5]
23. [WO04] Wensveen, S. and Overbeeke, K. 2004. Fun with your alarm clock: designing for engaging experiences through emotionally rich interaction. In Funology: From Usability To Enjoyment, M. A. Blythe, K. Overbeeke, A. F. Monk, and P. C. Wright, Eds. Kluwer Academic Publishers, Norwell, MA, 275-281. [7.5;7.5]
24. [WO02] Wensveen, S., Overbeeke, K., and Djajadiningrat, T. 2002. Push me, shove me and I show you how you feel: recognising mood from emotionally rich interaction. In Proceedings of the 4th Conference on Designing interactive Systems: Processes, Practices, Methods, and Techniques (London, England, June 25 – 28, 2002). DIS ’02. ACM, New York, NY, 335-340. DOI= http://doi.acm.org/10.1145/778712.778759 [7.5;7.5]