The bushing should float a bit in the holder so that it can properly align with the shaft. If you use epoxy, it won't float.
Is the bushing held by that washer and metal piece? If so, try adding a washer or some padding inside the clamped structure to tighten up the bushing a bit. A really good solution would be a felt washer that could retain some oil, or maybe just some string.
Also, be sure that the bushing spins very freely on the shaft and has proper lubrication.