HandMol: Coupling WebXR, AI and HCI technologies for Immersive, Natural, Collaborative and Inclusive Molecular Modeling

27 November 2023, Version 1
This content is a preprint and has not undergone peer review at the time of posting.


Except for isolated developments and specific software extensions, molecular graphics and modeling have historically been stuck at flat screens for visualization, mouse operations for molecular manipulation, menus and command line interfaces for controls, and single-user interfaces that only allow collaboration by streaming video hence limited to just sharing the view of the user operating the software. We demonstrate here how various technologies are ripe enough to enable much more fluent, immersive and natural human-computer interactions that in turn facilitate collaboration between human users, using affordable hardware through the internet and without even installing any specialized programs. For this, we introduce HandMol, a web app that exploits (i) WebXR for molecular visualization and manipulation in virtual reality, (ii) speech recognition coupled to a large language model to pass commands orally, (iii) speech synthesis for auditory feedback, (iv) WebRTC to communicate multiple instances of the tool without even requiring a server, and (v) external APIs to flexibly account for molecular mechanics, exemplified here with an endpoint running an AMBER forcefield for protein and nucleic acids and another running a DFT-trained neural network, ANI-2x, to allow exploration of conformation and some simple reactivity at high speed and accuracy. We show example applications to situations from daily work and education in chemistry and structural biology where HandMol can provide an advantage over traditional software: exploring and explaining molecular conformations and reactivity, docking and undocking small molecules into/out of protein pockets, threading molecules through nanopores, preparing systems for molecular simulations and for protein design, etc. We also present a brief study showing how users, even with limited or even no experience in VR, can significantly benefit from these kinds of technologies. As a draft prototype for the moment, HandMol is made available free of charge and without registration at https://go.epfl.ch/handmol, in (optional but greatly appreciated) exchange for feedback on usability and on features expected for this kind of tools.


virtual reality
augmented reality
chemistry education
future of work
molecular mechanics
virtual screening
molecular modeling
molecular dynamics
machine learning
artificial intelligence
molecular simulation
molecular dynamics simulations

Supplementary weblinks


Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting Policy [opens in a new tab] - please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .
This site is protected by reCAPTCHA and the Google Privacy Policy [opens in a new tab] and Terms of Service [opens in a new tab] apply.