This decision concerns an invention that relates to a video conference system that ensures maintaining eye contact with the users. The Board agreed with the applicant that a face detection system that locates a face in the display image and adjusts the camera such that they align with the eyes of the face ensures maintaining eye contact of the participants, thus technical. Here are the practical takeaways from the decision T 2306/17 of July 27, 2022, of the Technical Board of Appeal 3.5.04.
One obstacle to the widespread acceptance of video conferencing is the inability of video conferencing systems to promote socially acceptable eye contact. In most video conferencing systems, the camera is placed at the top of the monitor, so when a local user looks at the monitor, the local user is not looking into the camera.
The invention provides a system where the camera is positioned behind the screen (see, Fig.1). The screen alternates between a transparent state when the camera images the user and a light-scattering state where it projects a display image on the screen.
In order to ensure eye contact with the participant, the invention provides a face detection and tracking system to identify the face of the other participant on the display screen and align the camera behind the screen with the eyes of the participant (see, Figs.6-7).
Claim 5 of the Main Request
5. A video conferencing system (10), comprising:
a screen (12) selectively switchable between a substantially transparent state and a light-scattering state, where the screen includes a polymer stabilized cholesteric textured liquid crystal optical shutter;
a camera (14) positioned to view a camera image traveling through the screen;
a projector (16) positioned to project a display image at the screen;
a face detection system (18) to locate a face in the display image and align eyes of the face with the camera, wherein the camera’s position and orientation is aligned with the eyes of a face displayed on the screen, such that the camera has the same view as the eyes of the face being displayed, wherein the optical axis of the camera is perpendicular to a viewing surface of the screen and wherein the optical axis of the camera intersects the screen in a point positioned between the eyes of the face, where the camera is moved to align eyes of the face with the camera; and
a sync system (20) configured to cause the screen to alternate between the substantially transparent state and the light-scattering state, configured to cause the camera to view the camera image while the screen is in the substantially transparent state, and configured to cause the projector to project the display image while the screen is in the light-scattering state,
where the sync system is configured to dynamically adjust a ratio of time the screen is in the transparent state relative to a time the screen is in the light-scattering state responsive to an ambient brightness.”
Is it patentable?
In the first instance, the Examining Division alleged that the subject-matter of the claims lacked inventive step over cited document D1 (US 2005/0024489 A1). The Board considered document D5 (Shinichi Shiwa: “A large-screen visual telecommunications device using a liquid-crystal screen to provide eye contact”, Journal of the Society for Information Display, vol. 1, no. 1, January 1993, pages 37 to 4) would be a better starting point instead and identified the three distinguishing features, and the technical effect of these features:
4.4 The subject-matter of claim 5 differs from the disclosure of document D5 in that the former specifies:
(a) that the screen includes a polymer stabilised cholesteric textured liquid crystal optical shutter
(b) a face detection system to locate a face in the display image and align eyes of the face with the camera, wherein the camera’s position and orientation is aligned with the eyes of a face displayed on the screen, such that the camera has the same view as the eyes of the face being displayed, wherein the optical axis of the camera is perpendicular to a viewing surface of the screen and wherein the optical axis of the camera intersects the screen in a point positioned between the eyes of the face, where the camera is moved to align eyes of the face with the camera
(c) that the sync system is configured to adjust the ratio of time the screen is in the transparent state relative to the time the screen is in the light-scattering state dynamically and responsive to changes in an ambient brightness.
4.5 The board finds that these distinguishing features have the following technical effects.
A technical effect of distinguishing feature (a) is an increase in the brightness of the screen (in relation to a standard liquid crystal display as mentioned in document D5, Table 1).
The technical effect of distinguishing feature (b) is an adjustment of a position of the camera capturing an image of a local user of a videoconference.
The technical effect of distinguishing feature (c) is a maximisation of the display brightness while arriving at a sufficient signal-to-noise ratio of the captured images.
4.6 The board finds that the objective technical problems addressed by the distinguishing features are as follows:
(a) how to increase the brightness of the screen
(b) how to adjust a position of the camera capturing an image of a local user of a videoconference
(c) how to maximise the display brightness while arriving at a sufficient signal-to-noise ratio of the captured images
The Board then noted that the features contributed to separate technical problems and thus can be considered independently for the assessment of inventive step. However, it is adequate that one of the features is novel and inventive to render the subject-matter of the claims inventive. Therefore, the Board merely discussed inventive step of one of the features (i.e., feature b – the face detection system) :
4.7 According to the board, these are partial objective technical problems which would each have been separately addressed by the person skilled in the art (see Case Law of the Boards of Appeal of the European Patent Office, 9th edition 2019, I.D.9.2.2). This is because the first problem relates to properties of a screen, the second relates to positioning a camera and the third relates to digital signal processing.
4.8 The board finds that the person skilled in the art faced with the second partial objective technical problem would not have arrived at distinguishing feature (b) for the following reasons.
Document D3 discloses face detection but with the aim of identifying eyes in an image and shifting the position of the eyes in the image to create the impression of eye contact (see D3, column 3, line 14, to column 4, line 4). Document D3 does not teach adjusting a position of a camera capturing an image of a local user of a videoconference.
Document D7 discloses face detection to modify the appearance of an entire head to create the impression of eye contact (see D7, column 6, lines 7 to 33, and Figures 3A, 3B, 4A and 4B). Furthermore, D7 discloses extracting a part of a larger captured image to keep a videoconference user in the centre of a displayed image (see D7, column 6, line 34, to column 7, line 39, and Figure 5). Hence, document D7 teaches modifying an already-captured image rather than changing the manner in which the image is captured.
The board disagrees with the examining division’s view that document D7 rendered the feature “a face detection system to locate a face in the display image and align eyes of the face with the camera” obvious (see point III. above).
Document D7 discloses identifying a face in an image captured by a camera. In other words, this is an image of a local user (see D7, column 6, lines 7 to 11, and column 6, line 68, to column 7, line 39). However, feature (b) under point 4.4 above relates to face detection to locate a face in the displayed image. This image displays the face of a user at the other end of the videoconferencing system.
None of the other prior-art documents on file discloses feature (b) under point 4.4 above, nor does this feature belong to the common general knowledge of the person skilled in the art. The board finds that adjusting a camera position based on detecting a face of a user at the other end of a videoconferencing system provides a non-obvious alternative to the image manipulation known from the prior art.
4.9 Therefore, the subject-matter of claim 5 is not obvious in view of document D5 combined with any prior-art document on file or common general knowledge.
Therefore, the Board agreed that the subject-matter of claims involves an inventive step, and the application was granted.
You can read the full decision here: T 2306/17 of July 27, 2022, of the Technical Board of Appeal 3.5.04
Preston is a European patent attorney at BARDEHLE PAGENBERG. He handles patent applications in the field of electronics and communication technology, particularly hardware and software inventions related to wireless communication and entertainment devices.