If the speaker has a mic, disable it in Windows as an input option.
In my experience with my Bluetooth headphones, if I was to use the mic on it, it would disable audio to everything apart from the program using it.
Idk if this will help you but it's worth a shot.