Category: Advanced and Experimental with Serra

The Measurable Self

This interactive immersive installation explores the invisible processes through which artificial intelligence observes, records, and transforms human behaviour into data. By placing the participant at the centre of the experience, the project visualises the journey from a physical presence to a machine-readable profile, revealing how AI systems perceive people as collections of measurable information. The experience begins with a simple act of person detection. As the participant enters the space, the system identifies their presence and establishes them as a subject within its field of observation. This minimal interface reflects the first stage of computer vision, where an individual is recognised not as a person but as an object to be tracked.

The second stage introduces real-time line graphs generated through hand and pose tracking. Every movement is translated into numerical values and live visualizations, exposing how gestures and body language become quantifiable parameters that can be measured and stored. Next, a heatmap and motion feedback system accumulates the participant’s movements over time. Rather than representing a single action, the visualization builds a behavioural footprint, revealing patterns and frequencies that are invisible to the human eye but valuable to machine learning systems.The final processing stage shifts from data collection to computation. The immersive space fills with floating numbers, calculations, and fragmented datasets that surround the participant, visualizing the hidden analytical processes performed by AI. This stage represents the organization and processing of collected information before it is transformed into a usable model.

Once the computation is complete, the interface displays “100% Complete”, followed by “Subject Categorized.” After an extensive process of observation and analysis, the participant is reduced to a simplified numerical identity and assigned a predefined category. Rather than criticizing artificial intelligence itself, the project invites audiences to reflect on the mechanisms of contemporary data-driven technologies. It questions how everyday interactions are continuously converted into datasets for machine learning and training, and how these systems inevitably reshape human experience by treating people as collections of measurable variables rather than complex individuals

This week was dedicated to filming and editing the final project. We booked the 270-degree screen room and successfully loaded our project into the space. Overall, the process went relatively smoothly, although we encountered a few technical challenges along the way. Initially, we struggled to fit the resolution correctly to match the proportions of the wide screen. Fortunately, with the help of the technicians, we were able to configure everything correctly. Another issue we encountered was that our project had been built using POPs, while the computers in the room were running an older version of TouchDesigner that did not support it. Thankfully, this turned out to be less of a problem than we expected, and we were able to find an alternative workflow without significantly affecting the project.

By the end of the day, we had managed to record live footage inside the installation while simultaneously exporting windows directly from TouchDesigner. Seeing the project operating within the 270-degree space was particularly rewarding, as it transformed something we had been viewing on laptops into a much more immersive experience. Recording the interaction also reinforced Serra’s earlier advice about the importance of showing the audience how the installation is meant to be experienced. Capturing the interaction with the work made the concept much easier to communicate.

After filming, we moved into post-production. We divided the remaining tasks between the three of us to make the process more efficient. I focused on editing the footage and arranging the clips according to the layout and narrative structure we had previously agreed upon. Charlotte worked on the script and voice-over, while Ayca concentrated on writing the project statement. Although we were all responsible for different parts of the project, we remained in constant communication and made decisions together. Working this way allowed us to complete the project efficiently and reinforced how much trust and collaboration had developed within the group since our previous unit together.

This week was mainly focused on bringing together the different stages of our project and transforming the raw tracking data into a more coherent visual experience. Building on the work from the previous week, we expanded our MediaPipe setup by introducing heatmaps to represent areas of high activity. Rather than simply showing tracked points and lines, the heatmaps allowed us to visualize where movement was concentrated, adding another layer to the idea of the AI observing and analysing the subject. We also spent time experimenting with motion trails and refining the way movement data was displayed to make the visuals feel more dynamic and informative.

As we developed these elements, we realised that having access to data was not enough; we needed to communicate it in a way that felt believable and visually engaging. We spent time researching references on data visualisation, interface design and motion graphics, looking at how graphs, labels and different visual elements could work together to create the impression of a sophisticated analytical system. We also began working on the final stages of the experience, including the completion screen. Inspired by some of the references we found, we designed a sequence that displayed messages such as “100% Complete” and “Subject Categorized”, helping create the impression that the AI had successfully processed the participant.

During our tutorial with Serra, she once again stressed the importance of clarity and encouraged us to think carefully about who the audience was and how the experience would be understood. Following her advice, we decided to fully embrace the 270-degree room for the final presentation and restructured the project around that environment. We also developed the analysis stage using the POPs plugin in TouchDesigner, creating abstract particle formations that represented the AI processing large amounts of human data. This became the most visually overwhelming section of the project and contrasted nicely with the final stage, where the visuals became much cleaner and more machine-like. By the end of the week, we had established a clear progression from data collection, to analysis, and finally to categorisation. For the first time, it felt like the individual pieces were coming together into a complete experience rather than separate experiments.

This week did not have a major workshop or technical lecture. Instead, Serra focused on explaining the requirements for the final submission and showed examples of how students from previous years had structured their work. I found this really useful because I had been feeling quite overwhelmed by the number of deliverables and was still unclear about what was actually expected from us. Seeing examples of successful submissions helped me understand how everything could be presented and gave me a clearer picture of how to organise our own documentation.

The rest of the session was dedicated to project work. During a brief discussion with Serra, she once again stressed the importance of clarity throughout the project. Although our concept involving AI and prediction algorithms was technically interesting, she reminded us that the audience needed to understand what they were seeing and why it mattered. One suggestion she made was to film the interaction inside the 270-degree screen room. She explained that even a short clip showing someone entering the installation and interacting with it would communicate the purpose of the project far better than screenshots alone. This advice made us think more carefully about how the final work would be experienced and documented.

We spent the remainder of the session working on the next stage of the project, which involved generating heat maps and motion trails from the movement data we had collected. After searching through tutorials and experimenting with different approaches, we were able to produce some visuals that we were genuinely excited about. Seeing these first visualisations emerge from the raw data was encouraging, as it felt like the project was finally beginning to take shape beyond the technical setup and into something with a stronger visual identity.

This week we explored Reality Capture in Unreal Engine using Reality Scan. We began the session by selecting an object to record, and Serra gave us advice on how to achieve the best results. She stressed the importance of capturing footage slowly, ensuring all angles were covered and avoiding reflective or shiny objects, as these tend to confuse the software. I decided to scan my headphones, and after processing the footage we imported the model into Unreal, cleaned up the mesh and applied textures. Serra also demonstrated how the recorded object could be imported as a point cloud, which I found particularly interesting. Seeing the object represented as thousands of points immediately made me think about the possibility of animating point cloud values to create abstract and dynamic visuals.

Although I enjoyed the process, the workshop confirmed that photogrammetry is probably not something I would pursue extensively. I enjoy modelling assets from scratch, and I found the process of cleaning and correcting scans to be time-consuming and often inaccurate. While the technology is impressive, I personally prefer the control and precision that traditional modelling offers. However, I could still see potential for using point clouds and imperfect scans creatively rather than striving for realism.

Outside of the workshop, my group began developing the technical side of our project. Having previously experimented with MediaPipe during a TouchDesigner workshop, I spent some time brushing up on it before introducing it to the rest of the team. Once we were able to successfully track body motion, we started discussing which parameters would form the basis of our prediction algorithms and visualisations. We eventually settled on tracking hip, wrist and shoulder positions. By feeding this data into the system, we were able to generate a line graph representing changes in movement over time. Reaching this stage felt like an important milestone, as it marked the completion of the first technical stage of the project and gave us a foundation to build more complex visualisations from.

This week we had a short but enjoyable workshop exploring 360° video production in Unreal Engine using Off World Live. Using environments downloaded from Fab, we learned how to render equirectangular 360° videos directly from Unreal before encoding the output in Adobe Media Encoder. Although the process itself was relatively straightforward, it was interesting to see how quickly immersive content could be produced using real-time workflows. The session also gave me a better understanding of the technical pipeline involved in creating 360° experiences, which I found particularly relevant given my interest in immersive storytelling.

After becoming familiar with the workflow, I spent some time experimenting with different scenes and camera positions. Seeing the results in a 360° format highlighted how different spatial composition becomes when there is no fixed frame. It also reinforced some of the ideas I have been exploring in my thesis regarding directing attention within immersive environments. Despite being a relatively short workshop, it was one of the more enjoyable sessions due to how quickly we were able to move from setup to final output.

Our group also used the opportunity to further discuss our project. By this stage, we had begun settling on the narrative we wanted to communicate and started dividing the production into different stages. This helped us establish a clearer structure for the project and think about how the different screens and visual elements would interact with one another. Having a more defined narrative allowed us to begin planning the screen layout and think more carefully about how the audience would experience the progression of the video.

This week we received feedback on our group project and discussed both the creative direction and the technical feasibility of our ideas. Since our concept revolves around AI, data extraction and prediction algorithms based on everyday movement, a large part of the conversation focused on whether we would realistically have access to the tools and infrastructure needed to realise the project. We discussed possible tracking methods and interactive technologies, and Serra suggested looking into devices such as Kinect and Leap Motion for capturing motion more accurately than a webcam. This was particularly useful because it encouraged us to think about the practical side of the project before becoming too attached to ideas that might be difficult to implement.

Serra also emphasized the importance of visualization. Rather than simply collecting data, she encouraged us to think about how that data could be translated into something visually engaging and understandable to an audience. She stressed that the visuals themselves should communicate the story and make the relationship between movement, prediction and AI clear. We were advised to research artists working with data vizualisation, glitch aesthetics and experimental digital media. One artist that was recommended was Rosa Menkman, whose work explores digital artifacts, glitches and the ways technology shapes visual communication.

Reflecting on the feedback, I realized that our project needed to be driven as much by visual communication as by technology. It is easy to become fascinated with AI and data collection, but Serra’s comments reminded us that the audience ultimately experiences the visuals rather than the underlying code. Going forward, I want to spend more time studying how artists visualize complex information and how abstract systems can be translated into images and animations that tell a clear story.

This week we explored ND displays in Unreal Engine and discussed how similar technologies are used in large-scale digital billboards and public advertising displays. I found the session particularly interesting because it demonstrated how real-time graphics can be used outside of traditional film and animation contexts. The lecture began with a guided exercise where we followed Serra’s instructions to create a basic display setup and understand the underlying workflow.

Once everyone had completed the initial exercise, we were given the freedom to experiment with our own designs, animations and visual styles. This was my favourite part of the session as it encouraged exploration rather than simply following a tutorial. Being able to modify the display and test different animations helped me understand the system much more quickly than if I had only recreated the example.

I also used part of the session to discuss our project with my group members. By this stage, we were beginning to narrow down our ideas and were particularly interested in themes surrounding AI and data extraction. One concept we discussed was using prediction algorithms to analyse everyday human movement and behaviour, exploring how seemingly ordinary actions can become data that is collected, interpreted and used to predict future outcomes. Although the project was still in its early stages, the conversation helped us identify a direction that connected our interests in emerging technology, real-time media and experimental storytelling.

This week introduced several Unreal Engine virtual production tools. The session focused on how mobile devices can be used to interact with Unreal Engine in real time, allowing cameras and performances to be controlled from outside the software.

The practical exercise involved setting up Unreal VCam on our phones and connecting it to Unreal Engine. Although we encountered several technical difficulties during the setup process, we eventually managed to get everything working. Once connected, it was interesting to see how a phone could be used as a virtual camera within a 3D environment. The experience gave me a better understanding of how virtual production workflows operate and how real-time tools can be integrated into animation projects.

Outside of the workshop, I decided to pursue Brief 2 for the end-of-term project. While both briefs were interesting, Brief 2 felt more open-ended and offered greater freedom to experiment with different technologies and creative approaches. I had also become increasingly interested in TouchDesigner after attending separate workshops on the software, and I felt this project would provide a good opportunity to explore it further. Fortunately, two of the group members I had worked with during the Collaborative Unit were interested in a similar direction, so we decided to form a team again. Having had such a positive experience working together previously, I felt confident that collaborating would allow us to be more ambitious with our ideas and experimentation.

This week Serra introduced the unit and outlined the structure of the sessions for the rest of the term. We also discussed the idea of designing for a “user” and how this could influence the development of our projects. The unit offers two different briefs: one focused on expanded animation and experimental practice, and another exploring photogrammetry, digital environments and storytelling through fragmented representations of real spaces.

A large part of the session was spent discussing the expectations and deliverables for each brief. Serra encouraged us to think carefully about which brief aligned best with our interests and creative goals, rather than immediately choosing based on familiarity with a particular tool or workflow. We were given a week to decide which brief we wanted to pursue before beginning project development.

Serra also recommended working in teams where possible, as many of us had gained valuable experience collaborating during the previous unit. However, she emphasised that working individually was also completely acceptable. At this stage, I was still considering my options and wanted to spend some time thinking about which brief would allow me to experiment with new techniques while building on skills I had already developed.