The expansion of multimodality

From a macro theoretical framework, Kress (2015), analyzes multimodality within a social semiotic perspective, where communication is seen as a process of designing meaning through the selection and combination of modes, a conscious and motivated activity in which sign makers engage in creating new meanings. This author points out that the classic conceptions of communication, centered on audio-vocal and written language, are no longer adequate to explain current practices that emerge in a globalized and neoliberal world. From a micro level, Mondada (2019) based on ethnomethodology and conversation analysis, studies how participants organize their actions by using various multimodal resources in their interactions. The emphasis is on real-time interactions, where embodiment, materiality, and multisensoriality are integrated into the unfolding meaning-making. There isn’t a fixed hierarchical order of these resources before people start interacting; it all depends on the context in which the interaction happens (Mondada 2019).

On semiotic repertoires and their assemblages

The concept of semiotic repertoires emerges from sociolinguistic and semiotic theories that explore the dynamic ways individuals and communities utilize diverse communicative resources. These repertoires include language, gestures, visual symbols, material objects, and other meaning-making resources that people draw upon in specific social and cultural contexts. Rather than being fixed or confined to individuals, semiotic repertoires are fluid, situated and shaped by ongoing interactions within their environments. Scholars have highlighted how they are co-constructed and contingent upon the activities in which they are mobilized, in what is termed ‘repertoire assemblage’ (Kusters 2021). This process involves chaining resources in specific ways that are responsive to situational demands. Over time, with repeated use in similar contexts, these assemblages can become sedimented, creating patterns of recognizable practices. Furthermore, the assemblage process is not merely mechanical but is deeply influenced by both practical considerations—such as the availability and accessibility of resources—and affective evaluations, which reflect how people perceive and value these resources emotionally and socially. Kusters (2021) underscores the dynamic interplay between context, practice, and evaluation in shaping semiotic repertoires, suggesting that they are inherently relational and adaptive, responding to the needs and constraints of particular communicative situations.

Canagarajah (2013), emphasizes the concept of assemblage as a framework for understanding how social agents, semiotic repertoires, and material ecologies interact dynamically to construct meaning. He highlights that this distributed practice positions communication as a spatial repertoire emerging from activity, explicitly rejecting the privileging of vocal verbal language as a sole, superior, or isolated medium (p. 39). This view foregrounds the interplay of diverse semiotic resources in meaning-making. On the other hand, Pennycook and Otsuji (2014) and Pennycook (2018) define spatial repertoires as the connection between an individual’s life trajectory and the places where their linguistic resources are employed. Spatial repertoires are not necessarily pre-existing and brought by the individual to a situation but can be assembled in situ, collaboratively, as part of distributed practice. These repertoires extend beyond one’s current proficiency and should include all possible semiotic resources, not just linguistic ones. Furthermore, spatial repertoires should be understood as deeply embedded in the material environment and shaped by social networks. They provide an alternative to grammar in analyzing meaning-making and communicative success, involving various resources that people use in different communicative contexts.

These new notions of language make it possible to vindicate the forms of language born in the experience of nonverbal autism, as Amanda Baggs does when exposing, with the help of a voice synthesis device, the idea that language should not be limited only to verbal communication or the forms that neurotypical people recognize. For her, constant interaction with her environment is her language, a means of dialogue with the world poorly understood because it does not fit into conventional molds (Baggs 2007).

Finally, it should be noted that according to Mondada (2019), there is currently a debate between those who argue that objects and other materialities have agency in communicative interactions and those who argue that these do not have the same agentive status as humans but are mobilized in human action. In this study we consider fertile new materialism ideas that propose there is a fundamental shift in what the minds, languages, and individuals stand for in communication, considering them to be deeply materialized and connected to social networks and environmental ecologies. From this perspective, meanings, and thoughts are not understood as fixed entities, but as phenomena that constantly emerge through practices distributed among people, social networks, and material environments. Furthermore, this perspective emphasizes that cognition and communication are embodied processes, in which objects and space possess equal agency to influence human thought and communication (Canagarajah 2021). We think that the notion of assemblages of semiotic repertoires illuminates alternative thinking about language and communication in interactions with persons with nonverbal autism. Likewise, we believe that the conceptual and methodological developments of multimodal conversation analysis (Mondada 2019) can contribute significantly to the understanding of how multimodality allows us, through the lens of embodiment and materiality, to rigorously see how bodies, objects, and the environment are integrated into the construction of meaning at the micro-level of each situated interaction. These are fertile proposals to explore the richness of the interactions from the experience of nonverbal disability, highlighting the importance of multisensoriality in how people perceive and respond to the actions of others.

Goico (2021) offers a vision rooted in everyday interaction, where deaf young people and their hearing interlocutors construct meaning not through a pre-established language but through assemblages of semiotic repertoires that go beyond audio-vocal verbal language (Spanish) or viso-gestural languages (Peruvian Sign Language). The study shows how the construction of meaning emerges from familiarity with bodies, gestures, and shared stories. The repetition of the assemblages enables understanding, allowing sensory asymmetries to be overcome through resources that do not depend on the linguistic code but on context and shared experience.

In the Latin American field of knowledge and research, efforts have been advancing to understand communication in disability through critical, poststructuralist and decolonial perspectives that place lived experience at the center. Among these, the study by Guerrero-Arias at al. (2022) analyzes the relationship between speech therapy and labor inclusion of a person with aphasia from a critical theory of disability and post-structuralist approaches. The findings show that semiotic repertoires use in interactions integrate diverse resources that go beyond linguistic structures, such as the dynamic interaction between bodies, environments and materialities. However, because these repertoires do not conform to the norms of the work environment, therapy continues to focus on repairing language deficits to seek work inclusion.

Also, Gómez (2023) explores ontological, conceptual and practical proposals from which new ways of encountering children with nonverbal disabilities can be envisioned. In particular, she examines the possibilities offered by relational ontologies, difference-in-itself philosophy, post-structuralist, post-humanist, as well as new materialist approaches and concepts such as languaging, translanguaging, and assemblages of semiotic repertoires. The aim is to expand the Spanish-language literature on new perspectives on communication and language from which these processes can be re-signified, to understand them beyond logocentrism, and to place them in relation to the social, material, and spatial environment. Existing conceptual and methodological resources are synthesized to enable the appreciation of the richness of languaging practices that emerge in the experience of nonverbal autism, rather than viewing them through a pathologizing and restrictive lens.

In relation to autism

Rivera Amarillo (2004) mentions that the biomedical definition of autism always denotes a subject that does not respond to the conventions that delimit and place it as part of the species, a subject that deviates from what is healthy and therefore constitutes an abnormality. From this essentialist and ableist ontology, a person’s value is based on his or her capabilities to be productive. Thus, the failure to meet these expectations in societies driven by markets and productivity is diminished.

The questioning of these hegemonic conceptions of the human being, particularly those that privilege symbolic language, has been addressed by many academics, as Anna Stenning has pointed out:

The idea that symbolic language is natural and proper to humans, whereas other forms of communication are somehow debasing, depends on the idea that there is a singular good life for all humans that is determined, in all cases, by that which distinguishes us from other species. Specifically, in focusing on forms of communication that privilege symbolic relations, we ignore the much wider ecology of semiotic practices that connect us to other species (Stenning 2024, 154).

This approach highlights how these assumptions limit our understanding of the connection and interaction with the non-human world, creating fractures and distancing us from other forms of life. In this regard, Berenice Vargas (2024), a Mexican anthropologist and autistic scholar, argues for the need for ‘a collective conspiracy for the liberation of animals, humans, and the Earth, fully acknowledging our profound connection and relationality with the world that has been attempted to be taken from us’ (Vargas 2024, 252). Such assumptions include thinking that the mind is the exclusive locus of knowledge, that humans have mastery of their psyche and intentionality, and their agency is individual and voluntary (Schatzki 2002).

Anna Stenning (2024) discusses how the community-building practices of disabled people create spaces for learning and for rethinking language as something beyond a symbolic structure, emphasizing the sensory, embodied, and relational dimensions of communication:

…if we are willing to explore the communities that disabled people build in the real world, rather than focusing on institutional structures developed by an ableist society, we learn how we might embrace disabled ways of being in language through ‘sensory orientations, interdependence, mutual-aid, and world-building, carework, and the ways that time interacts with the bodymind and language (Stenning 2024, 123).

In this sense, instead of assuming the autistic person as an ambiguous being; marked by both lack and excess relative to normative notions of the world, who is not wholly recognizable, understandable, or contained by the binary structure of an I and another, we seek to understand the person by analyzing the singularity of her/his difference, reconsidering her/his body as a possibility, that challenges us to reconfigure the concept of human (Shildrick 2022).

Purpose of the study

The principal aim of this paper is to analyze the communication between a child, Ela, and her father, Gus, through the lens of languaging, alongside the notion of assemblages of semiotic repertoires configured within a multimodal and multisensory ecology. Rather than seeking generalizations, this study aims to provide an in-depth understanding of a particular phenomenon. (Somekh and Lewin 2005). Using this approach, communication can be seen as a unique and solidary encounter and that we already have the conceptual and methodological resources to understand communication in the full richness of its differences. It is based on a case study, in which it is assumed that ‘social reality’ is created through interactions situated in particular social and material contexts and in specific historical trajectories.

Participants

The study was conducted in a city in Colombia and focused on the interaction between a 6-year-old child named Ela, and her father, Gus. Ela was diagnosed from a biomedical model, with autism spectrum disorder and minimal verbal skills.

The recruitment process began with a contact with Ela’s family, during which researchers confirmed the family’s willingness to participate. Prior to signing informed consent, three visits were made to Ela’s home, during which researchers participated in play activities to establish rapport. Following these visits, a formal meeting was organized to obtain informed consent, the purpose of the research and potential risks were clearly explained. At this time, any questions or concerns of the family were addressed. Subsequently, a more detailed meeting was held to explain the phased research process and agree on specific dates. The first video recording was made during a routine daily activity, which made it possible to assess Ela’s comfort level with the camera.

Materials and data collection

During the fieldwork phase, six meetings were held to collect audiovisual material in different scenarios. These encounters included recordings at home, a recreational outing with the APA institution, an outing to the swimming pool, and visits to the park near Ela’s home. A total of 60 video recordings were collected during these sessions. At the end of each encounter, a space was provided to ask the parents questions based on aspects that emerged during the session, aiming to explore their perspectives on the recorded interactions.

To manage these data, a multimodal/multilevel transcription and an orthographic transcription were respectively carried out. The former was used to capture and analyze the full range of communicative practices in the interactions between Ela and her father. It was chosen because it goes beyond traditional verbal transcription by incorporating various modes of communication, such as gestures, facial expressions, body movements, and environmental factors, which contribute to meaning-making. This approach allows for a more comprehensive understanding of communication, especially in contexts where non-verbal or alternative modes of expression are significant. This system is particularly valuable for analyzing complex interactions and ensuring that all communicative resources are properly accounted for in the analysis. For the purpose of this article, an interaction in a social event was selected, in this case, an outdoor swimming pool session. This activity was chosen because it is an activity that Ela enjoys. The orthographic transcription was used to systematize the data obtained in the interviews (Hernández et al. 2014), with the parents, during which their accounts and opinions complemented the information about the interactions.

Data analysis

To analyze the data, we first transcribed Ela’s communicative interaction with her father using an adaptation of Lorenza Mondada’s multimodal transcription system. This system allows the annotation of the temporality and spatiality of gestures, gazes, sensory practices, and movements.

General conventions according to Lorenza Mondada (2018)

The embodied actions are transcribed according to the following conventions developed by Lorenza Mondada:

* ---> The described action continues in the following lines. Until reaching the same symbol ---->* .

>> The action described begins before the beginning of the extract.

--->> The described action continues after the end of the extract.

….. Preparation of the action (the action is starting).

---- The apex of the action is reached and maintained.

,,,,, Fading of the action.

fig Exact time at which a screenshot was taken.

# Displays the position of a screenshot within the turn/one time measurement.

ric The participant performing the embedded action is identified in lower case in the margin. In contrast, when the participant speaks, it is capitalized.

For the present study, we used specific symbols to identify each participant and their actions, which facilitated a detailed analysis of the interactions:

• Ela:
  • Δ: Actions performed by Ela.
  • Λ: Ela’s gaze.
  • ▼: Ela’s vocalizations.
• Gus (father):
  • ʘ: Actions performed by Gus.
  • φ: Gus’s gaze.

In addition, abbreviations were used for the body parts involved in the interactions:

• F: Foot.
• RA: Right arm.
• RH: Right hand.

The following is a guide that presents the different symbols within the multimodal transcription and the meaning of their spatial location within it to facilitate the understanding of the work done here.

Ela intra-acting with the water and her father

At first, Ela, and her father [People] arrive at the swimming pool [Spaces]. Once there, he puts on her swimming cap [Practices] and puts the jellyfish at Ela’s disposal [Objects] that are later thrown by him to the edge of the pool’s steps [Spaces] (Figure 1). Following this, Ela enters the water to pick up the jellyfish; during this process, Ela’s father sits on the edge of the pool facing away from her, and Ela, noticing this, picks up the jellyfish and goes to her father, walking on the first step of the pool.

Figure 1 

Semiotic resources that collaborate in the interaction between Ela and her father.

When she gets to her father, he asks, ‘Are you going to go in the water?’ gesturing to the pool. Ela leans on her father (see Figure 2) and walks down two steps, entering the water; her father picks up the jellyfish (see Figure 3) and throws it out, telling Ela, ‘You have to rescue them.’ All of the above corresponds to the following transcript:

Figure 2 

Ela enters the pool.

Figure 3 

Ela, Gus, and the jellyfish.

In this scenario, Ela actively engages with a sensory ecology that shapes her unique ways of being and intra-acting in the world. These ways of being are embodied and enacted, guided by her affective and subjective preferences. As Mondada (2019) argues, communication is not solely a product of verbal language; instead, it emerges through the materiality of the body and the surrounding environment. Ela’s actions, such as her gestures, facial expressions, and interactions with objects, reveal how meaning is constructed through the material aspects of her body and sensory experiences. The environment itself—objects, textures, and spatial arrangements—becomes integral to her communicative practices. Through these embodied practices, Ela uses semiotic resources that are both situated and affectively charged, demonstrating how the body and its material engagements are central to her communicative processes; in this activity, we see how people [Ela and her father], objects [the jellyfish, swimming cap], spaces [the pool, its edge, the steps, the water] and practices [putting on the swimming cap, throwing, and catching the jellyfish, exhaling underwater to produce bubbles, producing sounds] are assembled.

The intra-action between Ela, her father, and the pool water exemplifies how diverse semiotic resource assemblages facilitate meaning-making and action coordination. In this scenario, both human agents—Ela and her father—draw on a repertoire of semiotic resources, including verbal cues from Gus, the physical presence of a jellyfish toy, their gazes, and the materiality of the water. This spatial assemblage is oriented toward a specific goal: encouraging Ela to enter the pool.

Drawing on Canagarajah’s (2018) notion of spatial repertoires, this interaction reflects how meaning is not derived solely from pre-existing linguistic resources but emerges from the collaborative, in-situ use of a wide range of semiotic resources embedded in the material environment. Water plays a significant role given that it possesses several sensory qualities that elicit specific human responses. It has an agentive quality that contributes to our connection with its substance. Ela does not enter the pool spontaneously, her previous experiences allow her to have an experiential understanding that the qualities of water do not permit her to breathe when she is in it; therefore, she does not enter spontaneously; her father then uses the jellyfish as a semiotic resource of invitation for Ela to engage in the activity. When paired with her father’s verbal prompt and gaze, the material object becomes part of a distributed communicative practice that moves beyond linguistic explanations. This aligns with Canagarajah’s view that communication is co-constructed through the dynamic interplay of people, objects, and spaces.

The interaction is also shaped by what Canagarajah calls agentive material objects, highlighting how non-human elements can participate in and influence human interactions. In this case, the jellyfish and the water serve as essential semiotic resources that scaffold Ela’s decision to enter the pool. The jellyfish operates as a tangible link between Ela and her father, facilitating their communication, while the water plays an active role in shaping their actions. The water’s material properties—its texture, temperature, and physical resistance—influence how Ela navigates the pool, illustrating how spatial repertoires are deeply embedded in the material ecology of interaction. The physical coordination between Ela and her father significantly enhances the communicative process. As Ela leans on her father for support and their gazes align toward the pool, their shared bodily orientation reflects a deep trust built through their prior interactions.

This intimate coordination, expressed in Ela’s reliance on her father, illustrates the confidence necessary for successful communication. Drawing on Maturana’s concept of languaging, this interaction shows how communication emerges through verbal language and the embodied interplay of actions. In this intra-active process, communication is shaped by the dynamic use of multiple semiotic resources—body posture, spatial relationships, and material objects—working together to achieve their communicative objectives. Verbal language, in this context, is not privileged; instead, all modalities collaborate to co-construct meaning.

Multimodal transcriptions of the interaction reveal that, despite the absence of complex vocabulary or syntactic structures, the intonation in Ela’s vocalizations plays a significant role in the communicative event. This aligns with Canagarajah’s (2018) argument that semiotic resources do not merely supplement verbal language but actively shape and transform the communicative process. In this interaction, intonation, gestures, and the material presence of the jellyfish and water converge to construct a meaningful, shared experience between Ela and her father. The semiotic repertoire they deploy is not fixed but dynamically assembled in response to the situation, emphasizing the fluidity and adaptability of communication across multiple modalities.

This interaction underscores how the assemblage of semiotic resources within an embodied, multisensory context facilitates the coordination of actions and a deepened emotional dynamic between Ela and her father. This assemblage, rooted in spatial repertoires and agentive material objects, enables a shared experience that transcends the audio-vocal verbal (Spanish) language (AVL) and foregrounds the significance of multimodal, multisensorial material, and embodied forms of communication.

The jellyfish as part of multi-modal communication

After the sequence described above, Ela’s father initiates a new action by requesting her to retrieve the jellyfish submerged in the water, verbally encouraging her by saying, ‘enter the pool’ accompanied by a pointing gesture. This multimodal interaction, involving both speech and gesture, and the jellyfish taken from Canagarajah’s (2018) notion of spatial repertoires, shows how communicative resources, again, are not limited to language alone (AVL) but extend to the bodily, spatial, and material object and dimensions within the interaction. Gus’ gesture acts as a directional cue within this spatial repertoire, allowing Ela to visually orient herself to the specific area of the pool to reach the jellyfish. Here, emplacement comes into play as Ela positions herself with the environment, the water, and the jellyfish, transforming the space into an active part of the communicative process.

Ela’s physical response—jumping, waving her arms joyfully, and vocalizing ‘iiiii’— (see Figure 4) demonstrates multimodality (Mondada 2016) in action. Her communication is not verbal but deeply embodied, drawing from her entire sensory and motor repertoire to express excitement. The multisensorial ecology becomes apparent, with the perception and performativity of colors and textures of the jellyfish contributing to the coordination of actions. The vibrant colors of the jellyfish and its positioning in different locations in the pool also take on an agentive role in this communicative assemblage. They are not mere objects but active participants that shape Ela’s actions, emotions, and sensory experiences.

Figure 4 

Ela sits in the pool.

This complex collection of her father’s gestures, Ela’s embodied movements, the materiality of the water, and the jellyfish’s visual and tactile stimuli form a dynamic and evolving interaction (Mondada 2016). The jellyfish’s agency, reflected in its ability to attract Ela’s attention and guide her response, highlights the entanglement of human and material objects elements in meaning-making. In this sense, the jellyfish is more than a toy—it becomes a participant that creates the conditions for Ela’s joyful expression and engagement with her environment. Ela visually orients herself towards that spot and immediately moves closer to the area of the pool where her father threw the jellyfish, at the same time, she jumps and waves her arms with a happy expression on her face (See Figure 5), while vocalizing: ‘iiiii’. Given his life trajectory with Ela, her father understands and explains this emotional expression:

Well, she likes it, because she is using the water object and I saw it as a game, something she liked and you learn to recognize these expressions because the interaction is rather routinized, for example, when I gave her something she doesn’t like, she makes a face or spits, while when she is happy she does like this: “imitates Ela’s expression of joy” so you learn to recognize the gestures of what bothers her or pleases her [Papa-Gus].

Figure 5 

Ela and her father’s happy gesture.

Ela attempts to retrieve the jellyfish with her feet in the sequence described. Still, she fails due to their depth, creating a moment where embodiment, articulated by Mondada (2019), becomes critical for understanding the interaction. Here, Ela’s body and her movements shape the communicative action, extending beyond mere verbal or symbolic interaction. When Ela fails to retrieve the jellyfish, her father reorients the sequence by providing embodied support—using his feet to bring the jellyfish closer while offering a new verbal cue, ‘Ela, come’ (see Figure 6). This shift reflects the multimodal nature of communication, where verbal instructions, bodily actions, and material objects all ensemble into a shared interaction. In this sense, her father’s embodied response provides a scaffolding that allows Ela to re-engage with the activity (see Figures 7, 8 and 9).

Figure 6 

Gus points to the jellyfish.

Figure 7 

Ela searches for the jellyfish.

Figure 8 

Ela takes the jellyfish.

Figure 9 

Ela brings jellyfish to safety.

Mondada’s emphasis on embodiment points to how the intra-activity between Ela, her father, the jellyfish, and the water creates a dynamic of mutual influence and transformation. The fluidity of the water (as a liquid) and the sinking jellyfish shape how Ela and her father navigate the activity (see Figures 10, 11, and 12).

Figure 10 

Ela splashes the water.

Figure 11 

Ela feels the water.

Figure 12 

Ela looks at the toy jellyfish.

Figure 13 illustrates Ela attempting to retrieve the jellyfish with her foot, thereby modifying the manner in which the game has been previously developed. Following an unsuccessful attempt, she dives and tries again to reach the jellyfish (see Figure 14); however, she is still unable to achieve her goal. Ela then moves toward the side of the pool. In response to this situation, her father, who has been observing the interaction, decides to intervene and introduce a change in the dynamics of the activity. Through a subtle gesture, he invites Ela to collaborate with him in retrieving the jellyfish. This intervention not only shifts the focus of the interaction but also creates a collaborative space between them, transforming the moment into a shared experience (see Figures 15 and 16). This process exemplifies what Mondada (2019) describes as the performative nature of embodiment—how bodies are not just passive but are constantly reshaping and being reshaped by the material and social world around them. The water’s properties and the jellyfish’s sinking movement compel Ela to return to the water.

Figure 13 

Ela searches for toy jellyfish with her feet.

Figure 14 

Ela searches for jellyfish underwater.

Figure 15 

Gus invites Ela to retrieve the toys.

Figure 16 

Let’s go get the toys together.

In this sense, Mondada’s framework of embodiment and multimodality provides conceptual tools to understand how meaning is co-constructed through the interplay of sensory, material, and physical elements, all of which contribute to the communicative event.