GaSViC2

Gaussian Splatting based Virtual Conferencing and Collaboration

 

Topic#

Name of the Topic

5

Gaussian-splatting-based reality capture for VR

 

Eligible Partner (1) lead

 prof. Ing. Gregor Rozinaj, PhD.

Professor

 

Eligible Partner (2) lead

Pavol Jelenek

Executive Director

 

STUBA

ART4

gregor.rozinaj@stuba.sk

palo.jelenek@art4.sk

www.stuba.sk

www.art4.sk

 

 

Main objective

Develop an advanced 3D Gaussian Splatting (3DGS) module for the CORTEX2 platform that allows remote participants in form of avatars or real video streams to collaborate within highly realistic 3DGS scene.

Objectives

·       Develop and integrate advanced Gaussian splatting module to enable high-fidelity rendering of complex 3DGS scene, ensuring realistic visualization for the CORTEX2 platform.

We will implement the newest approaches of 3D reconstructions based on state-of-the-art Gaussian splatting algorithms to enable high-fidelity rendering of complex 3D scene as a digital twin. This will ensure realistic visualization, providing an immersive and responsive XR experience for all users.

·       Enable Remote Participant in the form of Avatar (RPA) with free movement to engage in real-time audio-visual interaction and collaboration in a shared realistic 3DGS scene.

The goal of this objective is to develop and integrate robust multi-avatar communication within the 3DGS scene using all necessary CORTEX2 modules, which results in enabling users to interact in real time through audio and visual channels, fostering effective collaboration in a shared realistic scene.

·       Enable Remote Participant with a web Camera (RPC) to seamlessly collaborate within the shared realistic 3DGS scene with other participants through virtual screen.

The goal is to enable collaboration between avatars (RPAs) and remote participants using video streams (RPCs) within a 3D Gaussian Splatting (3DGS) scene.

Executive summary

We propose developing an advanced 3D Gaussian Splatting (3DGS) module for the CORTEX2 platform to enable high-fidelity visualization of complex 3D scenes as digital twins. The module will facilitate real-time collaboration among avatars (RPAs) and allow seamless interaction in an immersive XR session. The module will allow connecting the remote participants (RPCs) displayed on a screen in the 3DGS scene. We will ensure full interoperability with the CORTEX2 SDK and VR hardware, optimizing performance for smooth and effective collaboration.

This module is highly important because it significantly enhances the CORTEX2 platform's ability to meet the increasing demand for realistic and immersive collaboration within 3D Gaussian Splatting (3DGS) scenes.

This module is unique in that it enables communication and collaboration within a Gaussian splatting environment. By combining high-fidelity visualization with real-time interaction, it allows users to seamlessly work together and communicate in complex, realistic virtual spaces—marking a first in XR and 3DGS technology.

The project will be carried out by a consortium of two skilled organizations, Slovak University of Technology in Bratislava (STUBA) and ICT company ART4, s.r.o. (ART4). STUBA brings extensive research experience and academic insight, while ART4 contributes its knowledge of industry and practical expertise in implementing Gaussian splatting solutions, ensuring a strong combination of academic knowledge and production-level development.