I think I see what's happening...you're supposed to hover over your selection and release the L-button, and then the selection activates. This weird behavior is just because I'm assuming you should click on things with mouse-mode on, and then upon releasing it toggles the alt-mode. Up to you if you ...

If you enable alt-mode in the quick panel (L-button) on Switch 2 (instead of up on the d-pad) and immediately release the L-button, the alt-mode is turned off instantly. If you move the cursor after enabling alt-mode inside the quick panel, however, it stays.

Up on the d-pad works well though...but ...