Posts By Ian Gibbins

Ian Gibbins is a poet, video artist and electronic musician living in South Australia. His poetry has been widely published in Australia and overseas, and includes four books, two of which are collaborations with visual artists. His award-winning poetry videos, video art and soundscapes have been exhibited to acclaim at festivals, installations, galleries and public art displays around the world. Until he retired in 2014, Ian was an internationally recognised neuroscientist and Professor of Anatomy at Flinders University, South Australia, having originally trained as a zoologist. For details see: www.iangibbins.com.au. For Ian’s videos, see: 
https://vimeo.com/iangibbins. For Ian’s audio work, see: https://iangibbins.bandcamp.com/

The Inevitability of Narrative

As humans, we are driven to narrative. It is almost impossible for us to experience a sequence of events and not attempt to ascribe some kind of narrative arc to it. What just happened? What does it mean? How did it start? What will happen next?

There is a considerable literature on the theory and practice of creating narrative in different contexts: fiction or non-fiction; in print or on stage or screen; or via any other medium. Narrative can be language-based, as in a novel, or non-verbal, as in a choreographed dance, or a combination of both, as in a movie. But the basic structure of narrative mostly boils down to a few key points: there is a beginning, middle and an end; something changes along the way; it occurs in some kind of contextual framework. If any of these key elements is missing, we, the readers or viewers, inevitably will try to fill in the gaps.

Over the last 20 years, neuroscientists and cognitive scientists have made significant advances in understanding how – and to some degree, why – the brain creates narrative. Much of this new research complements well the ideas of theorists and practitioners concerning the role of narrative in literature, cinema, theatre and dance. Indeed, many authors have integrated data across the sciences and humanities to build new appreciations of how and why narrative works. Key foundation texts in this field include The Poetics of Mind: Figurative Thought, Language and Understanding by Raymond W Gibbs, Jr (1994); On the Origin of Stories: Evolution, Cognition and Fiction by Brian Boyd (2009); and How Literature Plays with the Brain: the Neuroscience of Reading and Art by Paul B Armstrong (2013).

From a cognitive point of view, we can consider the construction of a narrative as coming in different flavours: for example, we can describe a series of external events, that is, things that others can also describe from their own experiences of the same events; we can create a story from our own remembered experiences that is essentially private, since no one else can have those same memories; or we can create a fiction, a story that no one has ever actually experienced, even though such a narrative necessarily contains elements that are relatable to the external world.

Autobiography, memory and narrative

In order to create a narrative, we must access several different elements of memory. There is no single phenomenon called “memory”. Rather, there are many forms of memory, some of which are so fundamental to neural processing that we are not consciously aware of them. The duration of different types of memory can range from a few tenths of a second (eg the early steps of processing visual or auditory stimuli) to a lifetime (eg learning to walk or knowing what a flower is). Furthermore, many forms of memory lie outside easy verbal description, eg how to button up a shirt; how you solved a crossword puzzle; how you felt at lunch-time yesterday.

The components of a traditional first-person narrative (“I did this, and then I did that…”) rely on what is generally known as “autobiographical memory”. This is memory of what you have done in the past. It is fully private, in that only you can access it. Unless you have a very specific type of brain injury, it is updated continuously as you consciously experience the world. There are several remarkable features of this memory system:

  • It is initially encoded by the same part of the brain (the hippocampus) that also encodes and keeps track of our movements through space and time in our local environment. 
  • This means that salient events are automatically linked to a time and place.
  • Some time after these events are first recorded into memory, they are transformed into long-term memory stores elsewhere in the brain where they are associated with other memories of varying degrees of relevance or significance: the nature of event itself, the weather beforehand,  the person who wasn’t there, the name of the movie, the music playing at the bus-stop…
  • Each time these long-term autobiographical memories are actively retrieved they are remade: in effect, they are re-remembered.
  • The neural processes involved in imagining a future autobiographical event are almost the same as those remembering a past autobiographical event. People with a damaged hippocampus not only cannot remember what they did in the past, they cannot imagine themselves doing something in the future.
  • As a consequence, autobiographical memory, most of the time, is hopelessly unreliable (there are other factors contributing to this, but that’s another story).
  • And as a consequence of that, virtually all narrative is a construction of unreliable memories. In other words, I suggest, all narrative can be considered to be first-person fiction (and that is yet another story…)

For the reader or viewer of a narrative, we automatically feed the story (such as it might be) through our own autobiographical memory processor. We identify the temporal sequence, the salience of each component, the relationships between the components, assign meaning to elements we recognise and make a guess at the meanings of elements new to us. Even the most abstract, abstruse, uneventful “narrative” will have a beginning, middle and an end, a context in which it is viewed, an emotional and cultural framework within which we will evaluate it. And so we create our own narrative version of “the story”, almost certainly incomplete when compared with the narrator’s intent, perhaps remembered only fleetingly, but more likely, generating a new entry of our own autobiographical memory (“Let me tell you about the movie I saw last night…”).

Selling the story – you don’t need much…

This ability for the viewer to make narrative out of minimal clues has been well recognised for many years. A famous experiment by Heider and Simmel (1944) showed how moving abstract shapes are commonly read as behaving like people within a strong narrative arc.

Similarly, short form texts such haiku, one-sentence poems, even Tweets (!!) can be commanding in their narrative intensity. Advertisers have known all this for a long time. Successful television advertisements must tell a story in as few as 15 seconds (“Look at this! See, it’s amazing! That’s why you need it… Buy one now! Get it here!!”). Indeed, some television advertisements have been recognised as stand-out examples of short-form video narration, for example, highly-regarded productions for Budweiser and Google.

https://www.youtube.com/watch?v=_ZhO57NBJm0

Most television advertising lacks such sophisticated production values, but even the most simple, hackneyed approaches can inspire a poetry video narrative. In my 42nds, the timing of the scenes and the amount of text per scene closely follows that of typical advertisements 15-45 seconds long. This video was commissioned for outdoor screening in a downtown shopping mall and has since been screened on other public locations mixed in with genuine advertisements.

I’ll finish with two examples of masterful, albeit unconventional, poetry videos, both of which have featured on Moving Poems previously, that incorporate many of the elements mentioned above. These episodic narratives subvert the tropes of commercial television whilst illustrating the highly mutable state of autobiographical memory: Profile by RW Perkins and Human Condition by Rich Ferguson and Mark Wilkinson.

The success of these videos relies totally on our ability to:

  • recognise the contexts of what is happening in each episode;
  • hold the episodes in our short-term working memory long enough to understand the relationships between them;
  • synthesise the meaning of the full narrative in terms that make sense to us, thereby embedding the external narrative in our own autobiographical memory, ready to reappear, perhaps in a subsequent narrative of our own. Which is more or less what has happened in this essay…

Video, Poetry and Translation

From Greek to English… We also have a version in Greek and Spanish.

I’ve always loved language and languages. I did Latin, French and German at school and I could easily have ended up studying linguistics in a slightly different universe. For me, poetry and experimental writing are fundamental ways to explore the limits of language: to try to describe what cannot be put into words, to find out what happens when language is stripped down to its essentials (whatever they are…), to discover how the visual, oral and aural aspects of language interact.

Combining video with poetry and experimental writing has been a revelation in this context. In a video, text can be dynamic, as it changes and morphs in multiple dimensions. Voices can be added, distorted, re-timed, presented in counter-point to each other and to on-screen text; they can even be made to articulate the literally unspeakable via increasingly sophisticated text-to-speech algorithms. And then there are all the possible interactions between the text of the video and its audio-visual content.

Over the last few years, I’ve been increasingly interested in how we deal with translation in poetry videos. I have had many videos screened in non-English speaking festivals and installations where there is usually a requirement for subtitles in either English or the native language. But sometimes, there is an absolute requirement for subtitles in the native language. I also have collaborated with non-English speaking poets which has required me become familiar with at least some aspects their native tongue. And I have even created a genuine bi-lingual video poem.

English and French as equal partners…

So how do we deal with multi-lingualism? How should the translations work? There is a large literature on the nature of translation, as well as the underlying neuroscience of bi- and multi-lingualism. But I have been strongly influenced by a couple of books in particular, both of which have been written by authors who have translated some of the most influential experimental poetry and writing :

Poetry videos offer a unique slant on translation that is not available for written text: videos give us the option to hear the original language, and, indeed, read the original written text itself as well as a translation. And we still have access to all the same audio-visual material and its interactions with the text. One corollary of this is that the translated text does not necessarily have to be as “poetic” as the original. It may not even need to be a complete translation, if the sound of the original and its accompanying imagery allow.

It is very common for translated poetry videos to have subtitles added in a similar way to any other video or film. However, conventional subtitles can clash with the visual aesthetic of the video. Nearly all my poetry videos have at least some of the text embedded in them, as part of the overall visual design. When adding translated text, I try to use the same fonts, layouts and designs as the original, so that the look of the video is not changed and the translation is seen as a natural component of the work. Another option is to use closed captions for a translated version. It does not look very elegant, but it allows the user to turn the translation on or off. A further advantage is that the text exists as a separate underlying time-stamped file (eg .srt format) that can be translated by a third party as required.

English and Spanish translation share a common design…

So now to the big question: how do we actually get the translation? Ideally, you are sufficiently knowledgeable in the appropriate languages to do it yourself. I can do that well enough for German or French, and I have made a genuine bilingual English-French video based on a poem of mine, Signature, originally published in the French journal Recours Au Poème. But although I have become familiar with some aspects of other languages, most notably Spanish, Italian, Greek and Swedish, I cannot translate from English into them. Instead, I rely on machine translation, good dictionaries and, when necessary, advice from a native speaker.

Machine translation…

My preferred machine translation system is DeepL which, when tested on languages I do know, performs better than Google Translate across the board, especially when the language becomes more idiomatic or figurative. The suggested translations can be checked and fine tuned in several ways. One is to simply back-translate the phrase to see if you get the same thing. Using a different system for the back-translation (such as Google Translate) can also be useful. Another good strategy is to replace some of the words in the original with synonyms or near-equivalents and compare how they get translated. If the translation offers options, I often look them up in the native language dictionary or thesaurus (eg Wiktionary) that shows how they understand the meaning of the word and how it should be used. And I always keep a grammar book at hand for the language in question to find out how it is structured and how things like tenses, cases, pronouns, adjectives, etc, actually function. These resources have been critical for most of my translations.

Sometimes, the translation simply will not work. An implied meaning in one language perhaps cannot be made in another, often because of the way the grammar rules operate. As is well known to translators, there are idioms and turns of phrase that do not cross languages or cultures. Experimental writing or visual poetry that relies on the intrinsic structure of words and their grammatical variations may be impossible to translate in any literal manner. What do we do then? We simply watch the video, listen to the ebbs and flows of words that even a native speaker may not understand, and revel in the uncertainty of it all.

Click here to see all Ian’s videos with translations.

This essay first appeared on Ian’s blog, which anyone with a serious interest in videopoetry ought to follow. —Dave B.