ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

Erzhen Hu
University of Virginia
Jens Emil Grønbæk
Aarhus University
Wen Ying
University of Virginia
Ruofei Du
Google Research
Seongkook Heo
University of Virginia
The ACM CHI Conference on Human Factors in Computing Systems (CHI 2023)
PDF | 30s Preview | Video | Talk

Support Object-Sharing Practices During Remote Meetings

People share objects for ad-hoc and collaborative purposes during work-related and leisure-related meetings.

  • During work-related meetings with colleagues, people often engage in discussions about product details, remote instruction, and ad-hoc demonstrations, as well as engage in small talk by sharing personal items.
    • Can we support remote collaboration on physical objects just like how we interact with digital UIs?
  • During leisure-related meetings with distributed families and friends, people may show handheld and background items to enhance social interactions and conversations and create memorable experiences.
    • How can we support these casual and ad-hoc object-sharing experiences, considering the space users can access to are bound to the small “talking head” windows?

Frequency and Behaviors of Sharing Physical Objects

We conducted a formative study (124 Mturk users) and found out that:

  • More than half of the respondents reported showing objects at least occasionally during work-related and leisure-related meetings.
  • For their sharing behaviors, we discovered that the majority of users simply hold the object up to the camera.
survey data of formative study

Challenges

There are various challenges associated with sharing physical objects during video meetings, including those related to certain user behavior (holding object to the camera) and system design.

  • Trade-offs between Context and Focus: Due to the constraints of screen space and the camera’s field of view, users often have to choose between a closeup view of an object without showing themselves (just like the shared digital screen) and the entire view that simultaneously shows both themselves and the object (though at the expense of the object’s details).
  • Perspectives: Observing an object from different perspectives requires asking the local user to adjust the camera or the object since the camera only captures a 2D image of an object from a fixed point of view.
  • Remote Referencing: Referencing objects often involves multiple rounds of verbal communication as the remote user does not have access to the object.
  • Privacy: Current videoconferencing tools only reveal human from the virtual backgrounds without exposing physical objects that they intended to share.
The need to constantly adjust the camera or object position and angle to share objects during remote meetings can be attributed to the synchronicity of meetings and the limited access to remote space. Sharing objects via live videos can create friction and result in a tedious process of framing and coordinating the objects or cameras. Alternatively, preparing these objects prior to the meeting can be time-consuming and may not be adaptable to unplanned situations such as impromptu questions from other attendees.

Why Digital Copies?

Let's take a moment to ponder the way we share virtual items - 2D windows, applications, or for collaboration tools like Google Doc or Miro Boards, people can be easily cross-referenced and accessed texts, images, or sticky notes by all users in the shared digtial space.
Imagine if we could digitize physical objects and make them just as easily shareable during synchronous meetings. So let's break free from the confines of video windows and unleash the potential of bringing physical objects into digital realm.

ThingShare UI

With ThingShare, users can quickly create digital copies of physical objects in the video feeds, which be overlaid on the user’s video feed for sharing in context (Person View), magnified on a separate panel for focused viewing (Task View), and stored in the Object Library for reviews.

  • Two Video Layouts for Balancing Context and Focus
    • Context Mode: equal-sized Person Views for ad-hoc and expressive sharing with object and human
    Focus Mode: minimized Person Views and a magified Task View with close-up details
  • Sharing Objects via the Person View: Based on the results of the formative study, we found that people often share physical objects via their video feeds in an ad-hoc manner, such as causally sharing their personal belongings or prototypes without focusing on the fine details of the object.

    • We enabled the Context Mode consists of (a) the Person Views so that it shows physical objects via its person space channel. The user can drag-off a digital copy from the person view for a quick, ad-hoc, and expressive demonstration. They can store digital copies of the in the (b) object library as (c) snapshots or videos.
    survey data of formative study
  • Sharing Objects via the Task View: Physical objects are brought to meetings not only for ad-hoc demonstrations but are also produced, modified, and reviewed afterwards, such as product reviews, hands-on training, and design sessions. Our formative study identified several challenges in showing objects in detail, especially during work-related meetings.

    • Focus Mode enables two main UI changes: (a) the person views will be shrunk to the left side and a shared task view (b) appears at the staging area, allowing the user to display different data formats of physical objects in more detail and collaborative manner.
    survey data of formative study

Interactions with the Object Library

We provide an Object Library (top-right corner) as a temporal space to store captured perspectives of physical objects. The person view and task view video windows and Object Library served as containers for digital copies.
  • By a simple drag-n-drop, users can capture the digital copies from any video window and store them in the Object Library.
    Store a snapshot
    Store a short video
  • They can also reuse the digital copies by bringing them back into any video window.
    bringing back a snapshot
    bringing back short video

Interacting with Digital Copies

Local users can create digital copies of physical objects in their Person View, which can then be manipulated and placed in the Person View, or moved to another user’s Person View, Object Library, or the shared Task View. This allows users to easily create digital copies of a physical object from various views, combine them with their own video, and separate digital copies from the background.
  • When the user hovers over the person view, all detectable items will be highlighted in yellow contours to inform the local user that they are shareable. A green contour will appear around the user’s selected object.
    Object selection and interaction (contours)
    Both handheld and background objects
  • With virtual background on, they can also selectively show and hide digital copies.
    • Now, imagine you're looking at your own video, When sharing a video, you can choose to hide certain surrounding objects, but still you intend to show something else, To support flexible control of shared and private space with the virtual background on, a user can use the "Show/Hide Object" command to determine if a selected physical object is visible to a remote user
    Show or hide objects with virtual background
  • Including pointing gestures in Task View
    • When pointing gestures are used, the hand doing the pointing may not be captured by the camera.
    As the instance segmentation approach detects and segments all the items apart out of the scene, including the hands (reference space) that are pointing towards an object (task space). We compose the hand with the object mask to maintain reference space. The task view allows for efficient gesturing by showing pointing gestures on the Task View. The view is also unflipped so that both users can see the detailed content clearly.
    Pointing gestures with hand
  • Annotations on digital copies
    • Annotate on the digital copies without having to hold the object.
    Annotation on a digital copy will be saved so that users can navigate back notes made on prior snapshots.

Discussion Highlights: Reflecting on the Challenges

  • Focus and Context: The Person View and Task View helped the users to share object with focus and context when needed during remote meetings.
  • Perspectives: Capturing non-live data (i.e., snapshots and short videos) in addition to live videos improved the ability to show multiple aspects of objects to remote users. Moreover, participants preferred the non-live options that freed their hands, as opposed to live video sharing which requires holding items.
  • Remote Referencing: Remote user can have fine-grained control over the stored or frozen digital copies by bringing a parallel copy back to their video windows for independent and free manipulation. They appreciated that they could quickly ask questions about the digital copy and point to the specific part for quick reference, as their activities within the video feed are fully translucent to another user.
  • Privacy: By selectively allowing objects to be visible regardless of the presence of a virtual background, users have flexibility to show objects in their local environment without compromising privacy.

Bibtex

ACM Citation Format