How to make learning color sensor fun?
Color sensors are small components with a surprisingly complex process behind them. Understanding concepts such as RGB filtering, light detection, frequency output, and sensor pins can quickly become abstract when they are taught only through diagrams or technical explanations.
What if learners could understand a color sensor by actually building and experimenting with one?
We created Martians, a gamified AR learning experience where players help a stranded robot on Mars repair its damaged color sensor.
Problem
Hardware concepts are difficult to understand when students cannot see what is happening inside them. For someone encountering electronics for the first time, these relationships can easily become a list of components and pin names to memorize rather than a system they understand.
User persona
Our target users are children aged 13+ who are already comfortable with digital technology and able to understand more complex concepts. They are currently taking basic robotics classes, which means they are familiar with robotics fundamentals but have little to no understanding of how a color sensor actually works.
Needs
I need to see how a color sensor works, not just read about it.
I want to interact with what I’m learning so I can understand it better
I need examples that connect the sensor to real-world robotics applications.
Pain Points
I know what a color sensor is, but I don’t understand how it actually works.
It’s difficult to imagine what happens inside the sensor from diagrams or text alone.
I don’t understand how each component contributes to the sensor’s output.
Behaviour
Primarily learns robotics through textbooks and classroom materials.
Responds better to visual and hands-on learning experiences.
Curious and exploratory; often asks “What happens if…?” questions to understand how components behave under different conditions.
Exploring solution
We realize that AR gave us an opportunity to explore in the realm of visualizations. We did a lot of iterations of the user flow together.
Our team did a crazy eight to build the narration of the story. Here’s me trying to hold the papers together lol!
The solution
Martian is a game where users learn about color sensor through trial and error. Using AR, players inspect and assemble the components required for the robot's color-sensing system. We decided to break down the TCS3200 color sensor into concepts rather than immediately designing screens.
Core flows
Meet the Robot
The experience begins with a short science-fiction story. A robot travels through space before an accident causes it to crash on Mars. When it wakes up, its color vision is malfunctioning.
- The introduction is presented entirely in 2D to establish the story before transitioning into AR.
- In AR, the robot is placed beside the workboard, giving users a consistent visual anchor throughout the experience.
Search for Components
Users explore their surroundings to find the missing components needed to repair the robot's color sensor.
- Placing components within the user's physical environment encourages exploration and makes the experience feel more interactive.
- Each discovered component triggers an explanation from the robot, introducing its function within the color sensor.
Assemble the Components
Users assemble the sensor components in any order. As each component is added, they can observe how it changes the robot's ability to perceive color.
- The interaction teaches through cause and effect, allowing users to understand each component by seeing its impact directly.
- Users can switch to the robot's point of view to observe how its color perception changes as the sensor is rebuilt.
Experiment with the Color Sensor
Once the sensor is assembled, users can scan colors from their surroundings and follow how the sensor processes each color step by step.
- The simulation responds dynamically to the scanned color, producing different processing steps depending on the input.
- Users can scan as many colors as they want, encouraging experimentation and learning through curiosity.
Designing for AR constraints
Designing Martian also meant designing for an interface where we could not fully control the environment. AR doesn’t have predictable boundaries like a traditional screen usually does. Because of that, we made preventive measurements.
Every edge case in the surface detection is accounted for, including detecting walls as surface.
But that being said, every kinds of floor surface is possible (even heads).
Electronic components that are tiny in real life are intentionally enlarged.
Feedback and iterations
We received feedback from experts in the AR field, and one insight stood out: a strong narrative gives users a reason to care and motivates them to stay engaged until the end. We also learned that AR shouldn’t be used simply to visualize objects in 3D. The experience becomes more meaningful when users can actively interact with the AR world and the objects within it. Photos from the feedback session are coming soon! I’m currently waiting for consent from our experts. :)
We also had the opportunity to share the app on the Apple Developer Academy’s exhibition, and we’re delighted to see that people of all ages are enjoying our game!
Reflections
Key Takeaways
Designing for AR Experience is very challenging and interesting. A strong narrative and visual cues makes people interested in our game (and booth!).
We have to make everything goes very seamless and smooth for the users.
Apple’s support for developing AR/VR project is amazing! For the first time, I learn how to use Reality Composer Pro and it’s very intuitive to use.