I think hand tracking is the way to go for this project, or at least its the best place to start for the interaction with the UI
I think trying to use controllers while cleaning or doing productive stuff would hinder rather than help so I am focusing the effort on getting Hand Tracking working
Here is my first implementation of the hand tracking / interaction using some images I generated using Stable Diffusion
I took the grab interactable prefab and tweaked it a bit to allow me to use it as an anchor for the image, so I can move it around my physical space
I think I am going to build this out some more and turn it into a module that I can use as a picture frame
