Thinc wanted visitors to physically push into a flexible rear-projection surface and see their input distort a projected “fabric of spacetime” in real time.

The core technical challenge was:

How do you measure depth and pressure across a large deformable surface, without installing visible sensors or mechanical hardware?

A large frame was built (approx. 6 ft wide) with a stretched Lycra-like fabric surface. A rear projector displayed a grid-based universe visualization, while a Kinect depth camera behind the fabric captured deformation depth.

To interpret input, I extended a CCV-based computer vision pipeline to support depth blob detection at multiple depth thresholds, allowing the system to detect:

• where a visitor was pressing
• how large the press region was
• how deep the surface was being displaced

Depth was used as a proxy for “mass,” allowing stronger presses to generate stronger gravitational pull in the simulation.

The prototype demonstrated that a large-scale tactile surface could be translated into stable multi-point interaction data, making the exhibit concept physically viable.

Thinc envisioned an exhibit where visitors could walk up, point at a massive starfield display, and “grab” a star from across the room.

But they insisted on a major constraint:

Sensors could not be visible, and the tracking hardware could not be positioned in the normal Kinect orientation.

This meant the Kinect would be mounted low in the exhibit station, facing upward—essentially using it backwards.

The hard part wasn’t detecting a person. It was:

• determining which direction they were pointing
• inferring a target point on a distant screen
• supporting repeatable gestures that felt natural to visitors

I built an openFrameworks prototype using Kinect V2 depth data and OpenCV-based segmentation.

Instead of relying on skeleton tracking (which breaks at extreme angles), I treated the visitor’s arm as a depth silhouette and extracted a pointing vector by detecting:

• the arm’s entry point into the sensor’s view
• the furthest extended point (hand/wrist region)
• horizontal direction from entry → extension
• vertical direction inferred from relative depth between entry and extension

This produced an inferred direction line. The planned “grab” gesture was defined as a rapid pullback motion (arm leaving the sensor field), which is easier to detect than finger pinching.

The prototype successfully demonstrated pointing inference using depth-only geometry, even with the Kinect placed in an unconventional position.

A second concept was to mount the Kinect in the ceiling, looking down at the visitor stations, keeping hardware completely hidden.

This was appealing architecturally, but created a major technical risk:

Kinect skeleton tracking is not designed to reliably interpret pointing gestures from a steep overhead angle, especially in crowded environments.

I built a second prototype attempting to use skeleton tracking under a near-overhead camera placement. A cursor system was implemented so the visitor could aim at targets projected across the room, then trigger selection by pulling their arm inward.

This approach partially worked but exposed major limitations:

• skeleton tracking instability
• arm distortion when visitors stood too close or at odd angles
• frequent loss of tracking under real-world posture variation

This prototype was still valuable because it clarified feasibility boundaries and helped define what would and would not scale to a museum installation.