Mr. Butler: An Augmented Kitchen Board
A tangible creativity support tool designed to bring mindfulness back into everyday cooking. Using gesture recognition, LED guidance, and real-time feedback, Mr. Butler reintroduces joy and skill-building to a hyper-automated world. Built with PyTorch, Mediapipe, OpenCV, and Arduino; evaluated using the Creativity Support Index (CSI).
Overview
Mr. Butler is a tangible user interface (TUI) designed as a Creativity Support Tool (CST) to guide users through cooking in a speculative future where AI dominates everyday life. Built with a laser-cut tactile board, gesture tracking, and LED feedback, the system scaffolds learning for novice cooks and encourages exploration without replacing user agency.
- Role: UX Designer & Developer
- Timeline: Sept 2024 – Dec 2024
- Team: 2 members
- Tools: PyTorch, Mediapipe, OpenCV, Arduino, Tkinter
- Focus: Tangible UI, Wizard of Oz Prototyping, Creativity Research
Background
In the year 2035, artificial intelligence automates nearly every personal task, from calendar management to meal planning. While productivity has soared, creativity has diminished. Cooking, once a creative and cultural act, has become rare.
Mr. Butler reimagines the kitchen not as a site of optimization, but as a space for personal expression. Inspired by CST frameworks, embodied cognition, and dual coding theory, the system supports mini-c creativity (learning) and little-c creativity (everyday improvisation) through physical interaction.
Research & Discovery
Method
A Creativity Support Index (CSI) evaluation was conducted using a guided omelet-making task. Participants included:
- Two students with busy schedules and little cooking experience
- One professional cook who enjoys cooking but lacks time
Questions were adapted from Dr. Sowmya Somanath’s CSI framework.
Key CSI Results
Statement | Avg. Score (1–10) |
Help me experiment with new recipes | 8 |
Make me a better cook | 7 |
Feel immersed using it | 4 |
Use it for unfamiliar recipes | 4 |
Use collaboratively | 2 |
Learning curve worth the effort | 5 |
Final CSI Score: 40 / 60
“Mr. Butler made me slow down and focus — I forgot how fun cooking could be.”
“It felt like Cooking Mama in real life.”
Persona
Amara, 29 – Consultant
Once a passionate home cook, Amara now suffers from decision fatigue and over-scheduled workdays. She craves a way to reconnect with creativity—without relying on AI to take over completely.
“Make it easy, but don’t take over. I want to feel like I made it myself.”
Brainstorming & Design Fiction
This project began with a speculative design fiction imagining 2035 as an era of over-automation and creativity loss. Mr. Butler was framed not as a solution to automate tasks, but as a collaborator in mindful action.
Guiding Design Principles:
- Low Threshold, High Ceiling, Wide Walls
- Support for Exploration and Iteration
- Multi-modal feedback for diverse learners
- Embodied cognition through physical interaction
Prototyping & Design Rationale
The prototype was developed using a Wizard of Oz technique and prioritized interaction over automation. Design requirements were based on dual coding theory, spatial cognition, and cognitive load minimization.
Design Highlights:
Users receive zone-based prompts using LED lights to reduce overwhelm and promote step-by-step engagement.
Gesture Recognition for Real-Time Feedback
Mediapipe tracks motion (e.g., whisking), and LEDs shift color based on speed and accuracy.
Feedback is visual, tactile, and textual (Tkinter GUI), supporting different learning styles.
- Adaptive Feedback
The system tailors prompts over time (e.g., suggesting slower whisking if users consistently go too fast).
Spatial Layout & Zones
The board is divided into clear task zones, aligning with cooking etiquette like mise en place.
Mocked Ingredients for Rapid Prototyping
Playdough ingredients were used to avoid food waste while enabling full interaction testing.
Technologies Used
- Hardware: Arduino Uno, Capacitive Sensors, RGB LED Matrix
- Computer Vision: Mediapipe (gesture recognition), OpenCV (object detection)
- Software: Python, Tkinter GUI, PyTorch (model scaffolding)
Prototyping Materials: Plywood board, colored clay
Evaluation
The CSI evaluation confirmed that Mr. Butler successfully supports creativity in structured tasks. Participants noted:
- It’s enjoyable and helpful for learning
- It made them slow down and think
- They wanted more flexibility and personalization
Strengths:
- Supports focused learning
- Makes the cooking process feel approachable and guided
- Aligns with user goals of exploration without automation
Limitations:
- Lacks open-ended creativity
- No multi-user collaboration
- Feedback not yet personalized to user skill levels
Results & Takeaways
- Developed a functioning TUI that encourages reflection and skill-building
- Demonstrated how embodied feedback can re-engage users with everyday tasks
- Validated CST principles through direct user feedback and CSI scoring
- Identified gaps in collaboration and advanced personalization for future work
“Mr. Butler isn’t here to cook for you — it’s here to cook with you.”
Future Work
- Integrate biometric sensing for personalized recipe suggestions
- Support left-handed users via remappable board layouts
- Expand recipe library and enable user-created workflows
- Add audio and haptic feedback for accessibility
Conclusion
In a world obsessed with speed and convenience, Mr. Butler offers a moment of pause—a space to create, learn, and reflect. It turns the kitchen into a playground for creativity, not efficiency.
Technology shouldn’t just do things for us.
It should do things with us.
Code Repository
This project was built with Arduino, Mediapipe, OpenCV, and Tkinter.
Originally developed in collaboration with Lindsay Gammon.