I tested the prototype on 5 participants. 1 participant comes from CS background, 2 from design background (architecture), and 2 from interdisciplinary background (HCI & cognitive science/management). Most of them have basic knowledge of machine learning or statistics.

The chart below visualizes the prediction accuracy of sample images in 3 rounds for 5 participants. The suggested round (model filtered) has the highest accuracy. But compared with the initial round, the accuracy of manually filtered one has significantly improved due to potential pattern learning from outlier suggestion function.

The survey and interview after prototyping session gave me more information beyond the prediction accuracy, participants expressed their finding, feeling and suggestion in this process, making design implication synthesis possible.

For alert interface, the second interface is preferred by participants because:
        - They can see the accuracy from it and understand the intention of deleting outliers.
        - It encourages them to think about further optimization for better performance.

For outlier filter interface, participants attributed the better performance to the simple and clear design.

Participants express stronger curiosity to Teachable Machine's mechanism than expected. While they are willing to learn more about it, participants try to keep an interesting distance from the algorithm, round 3 result disappoints participants with professional backgrounds while they are confident about their judgment in round 2.

Participants' feedback, especially their confusions, also brought some interesting questions for further speculation, which includes:
‍
        - Why Google puts “under the hood” into the advanced function?

        - How to balance human bias (users’ choice) and ML model bias (algorithm bias)?

        - How well could the filter system perform on unpredictable new datasets (generalization)?

They also provided valuable suggestion on the further interface iteration, which includes:

How to view the dataset:
‍
        - Row layout (integrated into the current TM upload section)
‍
        - Page view (added behind the upload section)

How to identify the marked outliers:
‍
        - In-site marking
‍
        - Outliers section for all groups
‍
        - Outliers section each group

How to backtrack:
‍
        - Archive section (backup the best-performance filtered dataset)

‍

The development of media technology and the rise of maker culture have significantly transformed how people engage in creative art practice. (Song, 2022)  On the one hand, the education process has shifted from traditional in-person courses to various forms of remote learning, a trend that was further accelerated by the pandemic. On the other hand, the increasing accessibility of art-oriented crafts has blurred the lines between professional and amateur practitioners, potentially reducing the depth of skill for craft practice (Song, 2022; Shiner, 2012). This shift has made remote education more feasible and accessible to a broader audience.

Craft, as a discipline of creative art, has experienced a similar transition. There have been numerous attempts to teach craft using different technologies, such as sensors, multimedia, and extended reality (XR), across various fields including pottery making, textiles, and origami. XR, in particular, has shown potential advantages for remote learning due to the immersive experience it offers and ability to overlay rich information onto the real world. However, our comprehensive literature review and immersive sessions with a ceramic instructor reveal that previous XR applications in craft education have not fully addressed the embodied nature of craft learning. Specifically, these applications often overlook the interaction between the body (particularly the hands) and the real-world context of materials and tools, the transfer of tacit knowledge for complex tasks (Jasche et al., 2021), and the social engagement in traditional apprenticeship (Wood et al., 2009).

Given these gaps, we wonder:
(1) How could tacit knowledge be made visible and learnable using XR technologies?
(2) How could XR technologies help learners learn from "critical incidents" (Groth, 2022) in complex craft processes full of "workmanship of risk" (Dhiman et al., 2024), as they would in traditional in-person apprenticeships?

To address these challenges, we adopted the Mixed Reality (MR) + AI technical framework. MR enables users to interact with real-world materials using their bodies, thereby maintaining the identity and authenticity of craft practices within a digital framework. Additionally, MR synthesizes multimedia elements to make tacit knowledge more accessible and comprehensible. AI enhances this experience by providing context-aware, adaptive feedback that mimics the interactive nature of in-person education through a question-feedback loop.

Based on both the theoretical and technical framework, we propose an AI-augmented Mixed Reality ceramic guiding system. Our system is built on Meta Quest 3 platform, focuses on wheel throwing—a fundamental skill for ceramic practitioners of all levels and a common starting point in introductory courses. The system allows users to directly interact with clay on a pottery wheel while receiving AI-augmented XR guidance. We structured our system with the theoretical framework of embodied interaction and iterated it by situating the system in real-world teaching sessions from the co-design process with instructors and learners.

Our development process leads to two AI-augmented components:
(1) a step-by-step teaching system equipped with multimedia interactions, such as voice commands and immersive videos, as well as hand gesture guidance through animations, simulations, and recognition.

(2) a real-time feedback system that utilizes rule-based corrections, computer vision-supported shape comparisons, and suggestions generated by large language models (LLMs).

These components can be integrated and adjusted according to the skill levels of different learners, offering a tailored educational experience.

Prototype 01: Moodie Assistant
Key words: Empathy, Emotional projection, Interpretation ambiguity
‍
Moodie Assistant depicts the algorithm experience as an emotional response to hallucination. This prototype takes the form of a voice assistant with a gauge indicating the hallucination level. It’s equipped with a series of remotes to enable users/audiences to express their emotional experience during the conversation with different modalities and granularity on hallucination control. Different roles, users and audience, have different emotional reactions in their interaction with the devices. The prototype’s embodiment gives us the medium to discuss the interpretation ambiguity.

Prototype 02: Whisper Web
Key words: Serendipity, occasional encounter situated in social context

Whisper Web explores hallucination experience as serendipity. The prototype embodies the form of a chatbot with personalized context “collection” to simulate LLM “training set.” The prototype reflects on the reaction when the hallucination provides deviated context implying different social relationships. Instead of intervening directly, the prototype’s embodiment leverages visualization medium to observe and document how the hallucinated info leads to the tension between human and conversational agents.

Prototype 03: Mindscape
Key words: Alchemy, turning hallucinated content into creative ideas, insights, and innovations

‍Mindscape explores the experience by seeking creative opportunities from LLM hallucination. The prototype is an XR application on the immersive environments platform to let users ideate and create an alternative world with the hallucinated LLM model, which focuses more on the brainstorming workflow and ideation/iteration. The prototype aims to investigate the effect of hallucinated-generated content on creativity. The embodiment mitigates reality constraints and augments imagination to the maximum.

Six participants were recruited through word of mouth for the pilot study. All participants are MSCD students/alumni. All participants have rich experience in design practice and are familiar with LLM-related applications/tools. They were paired into three groups, and three observation studies were conducted on three consecutive days. In the observation studies and the following interviews, one can listen to what other people “say” and watch what other people “do”. After analyzing the collected data with affinity diagrams and interactive visualization, a workshop was conducted in the following week for introspection and explorative participatory design to propose “solutions” as both design experts and user role.

(Note: Red: Recognition; Yellow: Interpretation; Blue: Relation)