Tech, Tools, and Terrains: Students Explore Immersive Film and Mixed Reality

The hands-on demonstration was part of the first of two Art Tech Bootcamps hosted at the CCAM earlier this month. Located at 149 York Street, the CCAM provides innovative arts programs and technical resources to all students, faculty, and staff. This fall’s installments of bootcamps, a popular, skill-based workshop format offered by Tsai CITY, were led by Berry, who introduced a diverse group of undergraduate and graduate students to immersive film and mixed reality technology. 

Berry, who graduated from Yale College in 2018, began exploring the artistic elements of technology as an undergraduate student majoring in Computing and the Arts, a field of study founded on interdisciplinary work and connection-making. As he tapped into a creative space shared by visual art and immersive media, he was faced with several questions: What does it mean to create in virtual reality (VR)? How do we effectively build art in virtual environments? 

Since working at the CCAM, Berry has continued to explore these questions. Besides delving into game development, educational applications of mixed reality or “XR” (an umbrella term for immersive reality), and research on various creative intersections, such as the connection between digital art and theater, Berry has been working on what he calls “democratizing VR:” making the technology more accessible through projects like his bootcamps.

“I have been thinking about how we can make VR accessible to people who aren’t coders, so that there’s more access to creating,” he said. “The bootcamps are focused on building those skills.”

Several curious students attended the first bootcamp, titled “Introduction to Immersive Film.” After a brief presentation, the session kicked off—and dove right into the tech behind 360-degree videos. Some students began tinkering with physical hardware—two different types of 360 film cameras (monoscopic and stereoscopic)—while others explored THETA, a mobile app that connected to the camera via bluetooth. All of the students had no prior experience with the technology, yet Seth Wenger, a graduate student who assisted with the session, advised students not to be intimidated by the technical details.

“The tendency to be surrounded isn’t new because of technology,” he said. “Panoramic paintings and images have existed for centuries.”

As students worked through the technical and visual aspects of filming with Wenger, Berry pulled students aside one by one to show what a completed film could ultimately look like in VR. Lifting the clunky virtual-reality headset up and over the students’ heads, Berry guided each of them through YouTube’s 360 film catalog, immersing them in the Amazon rainforest in one instance, and bustling Time Square in the next.

Then—back to reality. With the basic filming vocabulary in mind, the students were asked to split into groups and scatter around the CCAM with a tripod and camera. Their next task: to film. While the options were modest and often limited—they were on York Street, not New York—students still managed to produce a collection of short, varied videos: a still shot around the Yale Repertory Theater, a short tour of the CCAM, a stroll across a sunny parking lot.

 

In the computer, however, was where the real magic took place. After filming, the students congregated in the CCAM computer lab, where Berry imported their videos into the computer and showed them the artistic component of immersive film—what he described as “building scenes.” On Unity, a 3D game engine, Berry deftly dragged in a sample video, a still shot of a city’s skyline. The video appeared onscreen as a condensed, spherical rendering of the original scene, and by cutting a section of the panoramic scene—the sky and the skyline—and enlarging it, Berry created a new “layer” in his 3D world. The simple adjustment created space to fit other objects between the layers, compositing the 360 film with 3D models. 

As the students each took turns viewing the edited video through a VR headset, Berry mentioned further applications of the skills learned during the bootcamp. Animators, game developers, and filmmakers, Berry said, make the same artistic decisions and edits in their work every day. In addition to the creative application of 360 film, Wenger noted a few real-world applications: the School of Architecture is currently using 3D technology to model and view buildings, the Peabody Museum is using 360 cameras to document historical objects, and the same tech is even being used in surveillance systems around the world.

“It’s freaky and awesome,” Wenger added.

 

A week later, the second bootcamp, “Intro to Mixed Reality,” would take place with an almost entirely different group of students, but just like the first session, the learning was immediate. Berry began with a short presentation, describing the distinctions between virtual reality (completely digital), augmented reality (both virtual and real—“just think Pokemon Go”), and mixed reality: real and virtual with the added bonus of an interactive environment. And soon after, each student faced their computer, hopped into the Unity 3D game engine, and began following Berry’s instructions on building a 3D environment.

Each student started by creating the same basic 3D cube and flat terrain. But after a few minutes spent learning the tools in Unity, students’ screens were filled with shapes of varied colors and textures, strangely disfigured trees, and terrains filled with eccentric mountains and hills. After students were introduced to the Unity Asset Store, which contains free and for-purchase 3D models, the students’ environments further divulged from one another. Some creations were more conservative, such as a model car in a valley of pink hills, while others were far more extreme, like a forest of entangled tree branches overlapping with cubes, buildings, and human figurines.

As students experimented in their virtual environments, Berry further elaborated on how one could toggle an environment's lighting, create scripts to dictate how objects move in the environment, or connect their scenes to a virtual reality headset to view it for themselves. Certain decisions can be “artistic,” Berry added, while others could have a specific function, particularly in a game.

After the basics of Unity were established, the rest of the bootcamp was spent testing the creative limits of the game engine (hint: there were none) and using a VR headset to interact with a preloaded mixed reality game. Donning the clunky headset once again, Berry demonstrated how the students could teleport across the virtual world, shoot arrows onto a target, or even create an explosion of flowers. 

Then came the question the students were waiting for: “Anyone want to try?”

An unsuspecting passerby might have been alarmed to see students wearing cyborg-looking goggles, waving their arms around while awkwardly walking, crouching, and rotating—but after a day of skill-building, the brief period of ridiculousness was worth it. The students in each bootcamp left with modest creations, and, more importantly, they left with a new skill.

“There’s a lot of potential with this technology,” Berry said. “But what makes it interesting is when people from different views and backgrounds are able to work with it.”

Learn more about Tsai CITY’s programs here and discover the CCAM’s offerings here. Want to explore the intersection of art and tech yourself? Mark your calendar for an upcoming workshop on art and AI.