Starter keeps running

Sometimes when i turn the engine off, the starter continues to run. It doesn't turn the engine because the gear is not engaged, but I can hear the starter itself spinning. To make it stop I rapidly turn the key from off to start several times. When the engine is running, I can't hear the starter, or obviously I would not just let it run. This can't be good for it.

Second problem, sometimes the starter won't do anything when I want to start the engine. I get the loud click from the solenoid, but the starter does not spin.

Where do I begin? New solenoid, new starter, new key switch??? Thanks.