If you press the default key to Show/Hide the swatches panel (mine is F5), it gets activated and you can use the up & down arrows to scroll through the swatches. Press Return to commit, Escape to escape.
Unfortunately, the X key doesn't toggle between fill & stroke as long as the panel is active.
I couldn't arrow up/down through the swatches because my cursor was stuck in the Tint field. (I was showing the Fill/Stroke & Tint options in my swatches panel.) Once I hid those, it worked like a charm again. Tks!