OK, that sounds like the board is bad.
If you want to use that 3V board, you have to do a few things.
First off, you'll need a different battery harness and a 4 ohm speaker.
Next, you'll have to remove the black plastic shells from the dead 5V board and the white ones from the 3V board. You'll be replacing the white shells with the black shells you salvaged from the 5V board. You have to be careful doing this as you can damage the 3V board trying to get the shells off. I find some heat from my heat gun softens the plastic enough to make it easier to get them off, but it's still a tricky little endeavor.