Aditya Gunturu, Shivesh Singh Jadon, Nandi Zhang, Morteza Faraji, Jarin Thundathil, Wesley Willett, Ryo Suzuki. RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models. (SUI 2025) DOI: 10.1145/3694907.3765933
We introduce Augmented Physics, a machine learning-integrated authoring tool designed for creating embedded interactive physics simulations from static textbook diagrams. Leveraging recent advancements in computer vision, such as Segment Anything and Multi-modal LLMs, our web-based system enables users to semi-automatically extract diagrams from physics textbooks and generate interactive simulations based on the extracted content. These interactive diagrams are seamlessly integrated into scanned textbook pages, facilitating interactive and personalized learning experiences across various physics concepts, such as optics, circuits, and kinematics. Drawing from an elicitation study with seven physics instructors, we explore four key augmentation strategies: 1) augmented experiments, 2) animated diagrams, 3) bi-directional binding, and 4) parameter visualization. We evaluate our system through technical evaluation, a usability study (N=12), and expert interviews (N=12). Study findings suggest that our system can facilitate more engaging and personalized learning experiences in physics education.
Aditya Gunturu, Yi Wen, Nandi Zhang, Jarin Thundathil, Rubaiat Habib Kazi, Ryo Suzuki. Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology (UIST 2024)ACM, New York, NY, USA Page: 1-12. DOI: https://doi.org/10.1145/3654777.3676392
