Fish, floating signifiers and WhatsApp

Online multimodal communication seems to thrive when it's easy to use and experiment with. It's one of the more attractive features of WhatsApp as a platform. Still and moving images are easy to pull in, links work well, and voice messages provide a useful alternative to typing. And of course it's so handy just to click on to an audio or video call. Compared with old school synchronous chat, exchanges are more coherent. That's because the way in which you can quickly quote solves the problem of chaotic multi-threaded conversations. All in all there's plenty to research in WhatsApp chat and the smooth interplay of media makes it a gift for multimodal analysis. It may be a generational effect that I'm a bit slow on emojis, but I'm learning. Animated gifs on the other hand have tended to leave me totally bemused. Bemused until yesterday when my daughter just slipped one into the chat as naturally as you might raise an eyebrow in talking with a friend. And given that she's been studying this sort of thing I asked some basic questions and from her answers a whole new territory opened out. Searching gifs is a rudimentary affair and most of them could be described as floating signifiers in that they only become meaningful in their immediate context of use or within a particular speech community. Her partner's group of buddies use this fish one regularly in their banter - when someone has taken the bite. And it's therefore quite something looking back over a string of exchanges that effortlessly mix communicative resources - and all in the playful arena of casual everyday meaning making, the laboratory of human communication.

Mapping as data

A map can be very useful when you're lost - if you've got the right one, and if you can make sense of it. And pouring over maps can be quite compelling, too - imagining what it might be like to visit a place, or even to see the ways in which familiar places connect - connections that you had never before considered. But making maps is something entirely different - an artform in its own right, and its something I'm experimenting with at the moment. Trying to figure out different ways of presenting empirical materials has recently led me to think about writing otherwise, and in a number of recent publications Cathy and I have explored how storying our data, using different points of view may gesture towards multiplicity. We call this approach 'stacking stories', although admittedly we haven't yet managed to publish a full account of it. However, the shortcoming is that the story, whilst certainly capable of opening up other ways of looking, remains a predominantly linguistic medium. Cartography, on the other hand, presents different challenges, and although what to write is one of them, it only plays a small part. There has been some fascinating work on mapping as a way of tracing movement, and Abigail Hackett's focus on young children's movement around museum spaces is a great example. But after a recent research visit, I was tempted to try to map the remembered experience of the event. Not being particularly adept at mapping using paper and pencil, I looked for an online solution. It wasn't immediately apparent what would suit my needs, and there were a number of false starts. Eventually I settled on Inkarnate which is free, easy to use and has a pallette of Lord of the Rings-type icons. My original intention was to map felt experience, key moments and so on - the topography of the event, but the mapness of maps took over, and I ended up simply recasting where I'd been, as if the journeying was more significant than what happened (although, I note my stories often have a similar quality). But it's a beginning. Part of the problem is getting familiar with what this simple mapping tool can do; the bigger problem is what you might call translation. How can the territory of an event be mapped? At the moment I haven't got a clue, and maybe the map (or maps) might just end up being a supplement to the storying, but there's certainly potential here, and if not there's an engaging little hobby.

Discourse/methodology

Discourse and Digital Practices - the outcome of a lively symposium in Hong Kong, organised by Rodney Jones and his colleagues, is out now in the UK. In my chapter, I took some of the iPad data (also published here), and tried to relate my analytical methods to the more familiar practices of discourse analysis. Whether or not I succeed in doing this is, of course, for readers to decide, but to me it felt a little like retracing my own footsteps. I was trying to think through how discursive practices, media narratives, digital technologies, adult-child relationships, notions of literacies... and so on, all mesh together in the story-sharing episodes captured on video. In doing this, I probably focused too intently on the micro-analysis, applying a rather simplistic multi-modal framework to make sense of the interactions between adults, children and iPads. So although this generated interesting thoughts about pointing and touching, it bypassed embodiment and feeling altogether. But more importantly it excluded the intervention of researchers, ignored the representational nature of video and foregrounded human interaction. I was drawn into a comfortable world - one in which we can watch and describe 'fluctuating modal hierarchies' (Norris, 2004) from outside. But recently, and particularly in my collaborative work with Cathy, that idea that there are distinct actions, interactions and even episodes 'out there', waiting to be described, labelled, and interpreted becomes rather problematic. It's not that I've come to embrace a rather extreme form of social constructivism in which everything dissolves into being a sort of collective hallucination (although it must be said that the time spent looking at virtual worlds often made me question the distinction between the virtual and the actual, and sometimes to the extent that there seemed very little difference), but more that the work on multiplicities, socio-materialism and post-humanism that we've been reading prompts some rather challenging thoughts about what we do as literacy researchers - well, let's say social science researchers in general - with our dominant structuralist accounts of meaning-making and our almost doctrinal acceptance of social constructivism.

Modal aggregates

Norris's work on modal aggregates is a powerful way of conceptualising the iPad data that I've recently been writing about. Although her argument that multimodal perspectives destabilise the primacy of language in social interaction is not exactly new, the focus on action, as a unit of analysis, helps to highlight the shifting dominance of modes - how, in her words, modal hierarchies fluctuate. In the sorts of multiparty interactions that occur between children, adults and iPads, we can see how deictic gestures, at times dominant, give way to spoken language, onscreen visual movements, object-handling and so on. Accepting that overall meanings are always greater than the sum of these modal parts, this perspective helps to incorporate materiality and embodiment into the analysis in useful ways. There are some limitations, though. The analytical work that Norris (2012) engages in places humans as social actors at the centre of the interaction, thus following in the footsteps of earlier sociolinguistic approaches, but in doing this objects are cast as rather mute associates. For example, in Norris's data, a painting is moved (object-handling mode), pointed at (deictic gestural mode), and then talked about. But when scripted material objects - like iPads - are so deeply woven into activity, I think we need a broader perspective, one which shows how things (such as technologies) can generate, initiate, or participate in action. Perhaps it would help to focus on developing accounts of action sequences with different trajectories - the vibrating alert from a mobile phone that heralds an incoming text message, prompting some email checking, a phone call and so on, for example. Thinking about fluctuating hierarchies in modal aggregates might well be a useful way of approaching and understanding the emerging patterns of communication associated with new technologies. This could then lead to a more sophisticated account of how objects participate in social interaction, bringing what we read in Latour to bear upon our discussions of multimodality and discourse analysis.

The text vanishes

I found this year's LRA conference provided a rich fare. Some familiar things were being re-stated, some key work felt yet further developed and some radically new ideas seemed to be emerging. Cathy commented that the text itself received relatively little attention, and it does seem that with a growing interest in mobilties, and embodiment and affect, new constellations of sociotechnical and sociomaterial engagements are coming to the fore. Any signs of the text have now become ghostlike. Especially since now they have become open, provisional and often multi-authored, and in some senses ephemeral. Karen Wohlwend said something like 'I don't really like to talk about texts anymore, it seems that the kids are producing contexts.' I liked this, and although you could see in her work, and in that of many others, things that looked like texts - texts that formed and reformed - they were marginal things, like pencil shavings to the real sharpening of meaning that was occurring. The frozen fixed texts we used to pour over, the multimodal designs we read into them now seem to move backstage, as we watch gesture, movement and the claiming or enacting of meaning making spaces through mobility and countermobility. But what if the researcher's gaze is actually produced as much by data capture tools as it is by new representational practices and related theoretical advances? Does the availability of head-cams, spy-pens and flip cameras simply frame the segments of reality that we study in a different way? Perhaps though, we are learning more about the contexts that we always knew were there. Perhaps we have started to neglect text, or maybe it is slowly erasing itself as a fixed thing, a thing of privileged interest. When early emergent writing was a hot topic, we ran those paper-based artefacts under the photocopier and poured over the products while they were still warm. The contexts we tended to capture back then, if we had the time, the resource, and the inclination were more about talk, the stretches of verbal interaction that we caught on dictaphone tapes and then lovingly transcribed. As different then as new literacies might be, so are the possibilities for data collection. If we remain true to our reflexive project we are duty bound to think about what is newly acknowledged as well as what is omitted, erased or remains hidden, and how this influences the construction of knowledge. Did the play of power and the enactment of inequality, for example, just slip from our gaze or, if we are attentive, are those same forces simply seen in new ways and understood afresh?

What's on it?

It's getting quite common to hear about how technology can solve all our literacy issues in one fell swoop. But this story about using e-readers in Ghana as a way of raising literacy levels was inspiring, and an outcome of the work done by the Worldreader charity. So it begins to look like e-readers can be a relatively simple way of enhancing the reading experience of students of all ages. But yet it seems to me that there's two crucial things that are missing in this largely technocentric debate, and they are - perhaps inevitably - to do with the social dimension. Firstly, we know that something like a Kindle or an iPad is simply a platform for digital content. What counts then, once you've got over the love of technology, is the quality and values of that content. Is it a commercially-bundled curriculum? Is it worthwhile reading material, and does it really explore the multimodal (and other) affordances of digital content? Secondly, what happens when you insert this kind of device into everyday pedagogical situations? What sorts of new roles and routines develop (or could develop) for teachers and learners? There's a significant research agenda here, and we need some answers to avoid being blown away by technologists, or bundled into oblivion by commercial publishers.

Movement of signs

In the geosemiotics of traffic signs, this warning slips between being 'situated' and 'transgressive' (Scollon & Scollon, 2003: 22). In some ways it only works in the upright position. I've often wondered then, at what stage in its history a sign of this sort can be said to loose its authority? When it is fallen, rusted or removed, or simply out of date, what it is intended to signify begins to wear away, as it comes to represent something else - something for which it was not originally intended. Around where I live, signs often disappear. I am reliably informed that they have good scrap value, and in times and places of economic hardship this counts for something. Their meaning value is exchanged for economic value. Ironically, the one in the picture is just outside a scrap yard - it's probably been displaced by a reversing heavy goods vehicle, but at this point in time nobody has cashed in. Perhaps this sequence of events contributes to a sense of the ambiguities surrounding text and context, and how these shift over time: is this, perhaps, another version of the traffic of texts?

Putting things first

Joining the interdisciplinary group on space and place last Friday felt like taking time out. It was really fascinating to learn about the ways in which other academics are approaching different aspects of space/place in different ways. I learnt about ruin porn and object-oriented ontology (OOO), and some of the processes of artists - and somehow, in all that, it struck me that artists are often researchers and researchers are often artists. There's some really interesting blurring going on. In putting things first, humans often get erased - or at least all you see are their traces. Some of us wanted to re-insert people into their accounts, but presumably in ways that don't foreground or privilege them. All that was very stimulating, and the group project that we seemed to agree hovered somewhere between methodology and praxis. It's hard to say at the moment whether this signals a new direction for me, or whether it just sheds a different light on existing concerns. I'm content just to sit back and see what takes shape. So yesterday I took time out from taking time out, and spent the morning looking at ducks. Today there's only the duck sign left.

Different signs

Travelling quite often propels you into a very different semiotic context in which, at least in the first few days, you are unusually sensitive to what it is that you don't understand and those things which in theory, at least, should be completely obvious. You realise that it's far more than just language, or even one's ability to navigate the immediate environment. It is as if your whole communicative system has been derailed. My recent forays into the vocabulary of hands, gestures and pointing, and their place in everyday communicative encounters, has made me increasingly sensitive to the differences here in KL. There are, for instance, some very different ways of pointing and attracting attention. It's easy to be misinterpreted before one has even attempted verbal communication. Hands, arms and the rest also have deep historical, cultural and religious significance, as this Nataraj statue in the Batu caves illustrates.

Don't think twice (it's all right)

Ever since I came across this apparently innocent, 'off-duty' pic of our PM, I've been thinking about the issues it raises. You have to be of a certain age to connect it with the original album cover, but it's quite clearly a visual quote from Dylan's Freewheelin' cover which was released some 49 years ago. Visual quotes have become quite a theme in contemporary visual art and, of course, cinema too - Woody Allen is probably the leading exponent of this, but my personal favourite is Luis Bunuel's 'Last Supper' scene in Viridiana. (I'm not sure, by the way, whether to describe this as quote, parody or remix, but like citation, imitation and plagiarism in the written form, it's a phenomenon that is certainly worthy of academic scrutiny). But all this aside, what exactly is going on here? According to the Guardian newspaper, the photograph has been carefully set up by Tory party publicists as an explicit attempt to capture our leader in casual mode. That's the point at which all this becomes very subtle. So, assuming this is true - and there's no reason to doubt it - we have to begin questioning even the most seemingly innocent media image, that is to question its level of authenticity. The 'loving couple' in Dylanesque embrace may have just 'popped out for a curry' but the artifice suggests an attempt to create more....but more what? More of a sixties type counter-culture figure, surely not! Or is it trying to tap into something deeper, like a subliminal tilt at youthful romance by riffing off an iconic image deeply lodged in the memories of mature voters? If this is the case, then we might quite rightly recoil at an intention to subtley massage our political opinions. Actually, having said that, I'm not completely convinced  that is likely to alter my own attitudes to Cameron or to sway my anti-coalition political leanings, but maybe that's a different point. OK, well there's just one other avenue that opens up, and it's one that gives more positive credit to the photographer - a more innocent view. How clever, and how ironic to draw this visual parallel. All told, then, the image raises all sorts of questions; but of course the answer is blowing in the wind!

Do you love your book?

To my mind the main contribution of the multimodal movement has been to turn our attention to the broader terrain of meaning-making and to caution us against becoming obsessed with print and what print literacy can do. Some people have interpreted this as a turn away from print (or alphabetic) literacy, but the more intelligent commentaries accentuate how semiotic systems co-exist and interweave. Although this phenomenon is well-illustrated in digital texts, it has a much longer history. In fact you could argue that multimodal communication must by definition predate the advent of literacy. An interesting thought. But you need a socio-historical perspective to chart the rise of the influence of print literacy and particularly the ways in which this has translated into phrases like 'the power of reading', 'a love of books' and the whole notion of reading for pleasure. That particular configuration of power-love-pleasure is particularly influential in the discourses of literacy learning. Is alphabetic literacy powerful? I think you have to concede that it is, and that is likely to continue being so. Do you have to love books? Well not really. I've been brought up in that way, but I love lots of other things, too. I can think of things that might be a lot more useful and enjoyable on a desert island. And what about reading for pleasure? This is an idea which I find rather irritating. Not because reading isn't pleasurable, but because when I hear people say it, and particularly when I hear people talking about 'getting' children to read for pleasure, it seems to have an evangelical ring to it. It almost sounds as if pleasure in reading is a salvation - a turning away from the grosser pleasures of watching TV, playing videogames, supporting a football team and so on. Watch people reading - trains are good places to do this - they don't look as if they're being transported by pleasure (particularly noticeable on East Midlands trains!). No, I think when people talk about reading for pleasure,what they mean is what I call immersive reading. Immersive reading, like being in the zone in a video game, or 'lost' in a role play game is a rather special phenomenon - a sort of absorption in which one is in-between the narrative and the real world. It is socially and culturally significant (and it may also be psychologically important). It is what hooks us into virtual worlds, into good movies, into WoW, live drama and much more. It is our connection with the narrative universe and it just might be an essential part of being human, a way of imagining how things might be different - at different times, for different people, in different places. So I may just have transplanted narrative pleasure in the place of reading for pleasure....well at least that's a start. Let's acknowledge that the book is just one of many ways to get this absorption in narrative. But let's also keep open the question that narrative immersion may not be universal, too.

Searching limits

At the CSU technology symposium, we were once again alerted to the ways in which our casual searching behaviours may be shaped by search engines such as the mighty Google. This time it was Judy O'Connell issuing the warnings. So we learn to be sceptical of the JFGI approach. But since then I've got interested in something else about searching - and that's related to modality. I've been using the image above a bit lately, and have got quite interested in its provenance. A contact tells me that it shows the warehouse inventory process - possibly at a shipping line. Well that's OK, so far. But what I really want to do is search for the image and find out more about it. Of course Google (can I still mention it?),will allow me to search for an image related to a word, but it won't allow me to search with an image. Hence the modality issue. So I'm wondering if anyone's come up with a way of searching with an image? Maybe they have. Since some photo apps can now recognise your friends, image matching should be possible - and of course images are tagged and labeled. I wonder.

Uncanny valley

The Spielberg/Jackson Tintin certainly does 3D CGI animation well. But the movie hasn't gone down well with critics in the UK for two reasons: firstly, and this is an old debate, because it dumbs down Hergé (see Lezard here) and secondly, because of the CGI effect itself. I find the latter more interesting because I've been writing about visual realism, using Bolter & Grusin's (2000) notion of 'transparent immediacy'. They suggest that media-technology is advancing towards ever more life-like representations. So one critique of Tintin brings in Masahiro Mori's idea of the 'uncanny valley'. This idea suggests that the closer that representations resemble the real, the more that small inaccuaracies will derail us. Actually that didn't happen to me at all, but I'm glad to have learnt about the valley. You can read about that here.

Misspent youth

Collecting the football cards that came with Anglo Bubblegum stood me in good stead later in life. Redolent of that sugary fruity flavouring (a sort of guilty pleasure) each packet contained a card with basic details about a player, his club, his position and so on. I could never work out whether I prefered the information or the gum. They took it in turns. Maybe it was a rather dry strip of gum that ran out in less than a minute; maybe it was Danny Blanchflower or Jimmy Greaves; maybe it was just the taste you needed. But all that Anglo Bubblegum pedagogy served me well when the sixties came blowing in to the Midlands where I grew up. Not only did I bask in the popular music that seemed so new and fresh, I scoured the album covers for information. Who wrote those songs? Who recorded them? Who played bass and who did the cover art? I geeked out on all the information, always keen to identify the key players and to trace the influences that had helped shape what I liked. Nothing much has changed. What might seem like a misspent youth was actually a substantial investment. A shift in the field to the world of theory and research is actually quite a small shift. I'm still listening out for 'new tunes' and trying to understand how they came about (who they were listening to when they wrote them and all that). Surely that's the starting point of a good literature review - creating your own map of the territory. And to think it all began with bubblegum!

First Skype call

In affluent and highly-digitized societies childhood is infused with technology. Digital technology has to be factored in from the prenatal stage when the first images of a baby are scans on screens, circulated by mobile phone and posted on social networking sites. And for the newborn, not only are the toys digital but the everyday practices that they participate in with adults are similarly mediated. I'm slowly building up an archive of images - here's Dylan at six months doing his first international Skype call. And that's just the beginning!

Robots can dance

Will robots ever become part of everyday life in the way that computers have? Well, I suppose it depends on what counts as a robot. I'm looking for something robotic for our futures event in July and came across these little ISO-bots. They don't look great, but they can be controlled with a nunchuc which is pretty impressive. If your associations with basic robotics are anything like mine then they're about turtles and roamers and they become a sort of conceptual exercise in programming. Skip the alpha-numeric stuff and go straight to movement sensors and gestures and you have a whole new kind of robotic control; something akin to an out-of body experience! I looked for a good clip and found the one above. I'm not sure how these ISO-bots are controlled (I think its the standard controller), but the choreography is interesting. The guys' reactions are at least as good as the dancing robots! If you want more go here.

Images

I love Michael Constanza’s video of Hope Sandoval on this YouTube clip. The dreamy quality goes so well with the track. It’s interesting how readily music and image combine in new media. I'd love to write a bit more about music; I had so much fun doing that chapter in the Web 2.0 book. And image has been very much on my mind of late. So for my latest project I’m looking for a reliable way of streaming live video. I’ve been experimenting with U-Stream which does the job quite well. I also learnt today about Screencast-o-matic which seems to do everything I used to do with Jing, but not what I want to do on the project, but I’m looking into that one, too.

e-reading the news

The newspaper is delivered silently in the dead of night. There is no rattle of the letterbox, no picking up, no folding and refolding. I can't use it to swat flies with or stuff into my wet boots as they dry. There's no stack of yesterdays in the living room to raid for wrapping broken glass in, or for laying a fire or helping it draw. It's wirelessly delivered to my e-reader. And yes, it's because of all this I recommend a 10 day free subscription to any student of literacy in order to better understand how habits and material affordances interweave with our nostalgia for certain media. The expeience is qualitatively different. It has to be. But the news is arguably the same. Navigation requires a different logic and one in which I'm still a novice, shaving doesn't interupt my reading (I just turn it on voice) and if I'm reading outside a gust of wind is not a problem. I can check the details online at a single click. Yet for all its newness I'm disappointed with the reduced depth of modality. The colour supplements are grey and the illustrative content paired to the linguistic bone. So e-readers represent an interesting moment in technoliteracy, one in which convergence is always a promise and divergence a commercial reality. Ergonomically well honed, easily portable and with glare reduction to die for the Kindle has so much going for it. But then it's not particularly versatile. Although our technology is getting light and more mobile, you still need to pack your bag with all sorts of devices each one of which probably has the computing power to manage all those separate functions.

DJs go digital

töken experience from yöyen munchausen on Vimeo.

Well of course they went digital years ago - at least as soon as CD-Js took over from turntables. I first saw this when Shuiab Meacham and his kids rocked the Crucible in Sheffield. Apart from obligatory glances in the rearwiew mirror (making CDs sound like scratched and pitted vinyl) and nostalgia in the form of the decks-are-best crew, DJs have moved on with the technology using what the boffins come up with and the sound companies can sell. DJEmmaLou told me about the touchscreen interface in the video which takes DJ-ing into a new space. There is some talk of gestural music interfaces, but at the moment all we have is haptics (here). In my imagination the truly gestural DJ will also be a dancer whose movement in space triggers the sound we hear. Now that would be a hard act to follow!

Simple model of reading

I've been thinking about a simple model of reading that can be applied to texts of all kinds - books, new media and virtual worlds. There's only 2 aspects to the model (although that's a bit of a cheat, because they all break down much further). They are: 1. operating the interface, 2. engagement. First of all then, operating the interface in the typographic world involves all that book knowledge, directionality and decoding stuff, whereas the analagous skills in online texts are sometimes the same and sometimes different with large doses of clicking and pointing, dragging and dropping and so on. The important point about operating the interface is that for a skilled user it becomes naturalised and usually takes place beneath the level of consciousness. That frees up attention for the second aspect engagement. Not, of course, that they should be seen as sequential operations or to suggest a developmental path for learning. Engagement is all about meaning-making and at the moment I see it as having 3 dimensions. The first is interest, because without that we won't have the energy to give the text enough attention. The second is point of view. Somehow or other an interaction between our point of view and the point(s) of view in the text needs to happen. In other words we need to negotiate the meaning. Finally, we have belief. Seems a bit strange, I know, but after reading about virtuality (I recommend Laila Shereen Sakr) I'm starting to think that the textual meaning is in some senses virtual. We have to believe in it to engage with it. That doesn't mean we have to believe or accept the point of view or the message, but we have to believe the text to bring it into being. More on this, no doubt, in future postings. Stay tuned!!