We run courses focused on virtual reality, augmented reality, interaction design, and 3D user interfaces, offered to graduate students at Cornell Tech.

INFO 5358 / DESIGN 5358

3D Interaction Design

Designing products and experiences for head-mounted displays, often involving novel inputs such as hands and eyes, introduces new challenges for designers when compared to traditional devices and user interfaces. This course presents the theoretical and practical foundations for designing 3D User Interfaces (3DUIs) for virtual and augmented reality applications. The course will highlight the differences between 2D and 3D interaction design, introduce design principles, best practices for prototyping, and case studies.

Students will be tasked with designing, implementing, and evaluating interaction prototypes of different fidelities for 3DUI tasks such as selection and manipulation, travel, and system control. Product design case studies incorporating head-mounted displays, wearables, and novel input devices will be introduced in the context of 3DUIs.

This course is open to students from all Cornell Tech programs; from design programs to computer science. The theory and low-fidelity prototyping skills acquired in this course can be applied to design, engineering, and product management in the field of AR/VR.

Instructor: Harald Haraldsson
Term: Spring

Cornell Tech students who are interested in high-fidelity 3DUI prototyping and software engineering are encouraged to consider independent study at the XR Collaboratory, see FAQ for more information.

  • Required textbooks:

    • Joseph LaViola Jr. et al. 3D user interfaces: theory and practice. Second edition. Addison-Wesley Professional, 2017.

    Other readings, such as research papers, will be assigned throughout the semester.

    • This course is open to all Cornell Tech students. No coding or prior tool experience is required.

    • Enrollment priority is given to students who have completed INFO 5340 / CS 5650: Virtual and Augmented Reality

  • The lecture content in this course partially overlaps with CS 5678: 3D User Interfaces (not offered in Spring 2025). However, this course focuses more on design and low-fidelity prototyping, and less on advanced technical implementation, especially for homework assignments and the course project. Students who have already completed CS 5678: 3D User Interfaces are not permitted to enroll in this course.

  • Topics include:

    • Foundations of 3D User Interfaces

    • Perception, Cognition, and Ergonomics for 3D User Interfaces

    • Human-Computer Interaction and 3D User Interfaces

    • 3D Input & Output Devices

    • 3D Interaction Techniques: Selection and Manipulation

    • 3D Interaction Techniques: Travel

    • 3D Interaction Techniques: System Control

    • Design Strategies for 3D User Interfaces

    • Evaluation of 3D User Interfaces

    • The Future of 3D User Interfaces

    • Case Studies

    • Homework & Project: Designing, implementing, and evaluating 3DUI prototypes

    • Quizzes: In-class quizzes from lectures and readings

    • Participation: In-class activities, attendance etc.

    • Demonstrate understanding of concepts from human-computer interaction in the context of 3D user interfaces

    • Demonstrate understanding of classic 3D interaction techniques and current trends, both in academic research and commercial product design

    • Demonstrate understanding of the unique challenges and opportunities when designing products and experiences with 3D user interfaces

    • Demonstrate the ability to design 3D user interface prototypes of different fidelities

Spring 2025 - Selected projects

class-01.jpg
guest-lecture-02.jpg
guest-lecture.jpg
guest-lecture-03.jpg

INFO 5340 / CS 5650

Virtual and Augmented Reality

This course presents an introduction to virtual and augmented reality technologies, with an emphasis on designing and developing interactive virtual and augmented reality experiences. The course covers the history of the area, fundamental theory, interaction techniques, and specific application areas.

Concepts from the contributing fields of computer vision, computer graphics and human computer interaction are introduced in the context of virtual and augmented reality. Students are tasked with creating their own virtual reality application as a course project.

Instructor: Harald Haraldsson
Term: Fall
Course website

Assignments

  • Geometry

  • Meshes

  • Rendering

  • Locomotion

  • Interaction

Course Project

The course project focuses on the design and development of a custom virtual reality design tool, inspired by design applications such as Gravity Sketch and Arkio. Students are tasked with implementing a set features with the purpose of enabling their users to design a car in VR, using their tool.

Featured student project - Fall 2023
Connor Pugh, Gural Nuriyev, Sidharth Vasudev, Ruslana Yurtyn

Color picker feature

Color picker feature

Mesh manipulation feature

Mesh manipulation feature

Scaling feature

Scaling feature

Undo/redo feature

Undo/redo feature

    • Move object

    • Rotate object

    • Scale object (per axis)

    • Select single object

    • Select multiple objects

    • Mesh manipulation

    • Create object

    • Set shape for object

    • Change color of object

    • Delete object

    • Duplicate object

    • Undo / redo

    • Grab move locomotion

    • Visualize controller state

    • Help and documentation

CS 5678

3D User Interfaces

This course explores the field of 3D User Interfaces (3DUIs) and interaction design for virtual and augmented reality applications, including theoretical foundations, design principles, and best practices for high-fidelity prototyping.

Students are tasked with designing, implementing, and evaluating their own interaction techniques for 3DUI tasks such as selection and manipulation, travel, and system control for head-mounted displays.

Instructor: Harald Haraldsson
Term: Previously offered 2020 to 2024, see INFO 5358 / DESIGN 5358 for 2025 and later.
Course website

Assignments

  • Balloon Selection

  • Go-Go

  • Pengufly

  • System Control

Course Project

The course project focuses on designing, implementing, and evaluating interaction techniques for 3DUI tasks such as selection and manipulation, travel, and system control.

Featured student projects
See course website for more projects

Spring 2024

Spring 2022