Put a new starter on and it doesn't disengage. Is there a separate relay and solenoid, or are they the same thing(assuming it has to be that).

 

That is really weird. I would think the motor has a defective solenoid. Seems like you got a bad starter.

My brothers boat did that once and he didn't realize it was still engaged. Drove the boat and turned the starter into a molten hunk of metal!