2 simple things:
#1 the plastic those battery holders are made out of is extremely flexible. You could easily flex it far enough and push the battery out from the bottom. Again, there is barely a tiny lip holding the battery into the holder anyway and you have access at the bottom to push the battery forward and out.
#2 Being a PS2 5V, I would have removed the board for proper and better inspection of the capacitors on the board- and we wouldn't be worried about clearance.