On the 480 truck the rotation occurs as the wheel rotates on the axle. The hole in the truck side frame is there to support the truck, it is not a bearing. On the fast angle wheels, of which I have little knowledge, are the wheels pressed on the axle and the bearing is rotation between the end of the axle and the side frame? If this the case I would say the fast angle wheel sets are not interchangeable into the 480 truck.
In looking at your picture it looks like the flange is well worn. If this the case it could be part of the problem. If the flange is worn the bore of the wheels may also be worn. I have also found that bent axles can be a cause of derailments on a switch. I have changed a lot of bent axles. I have never figured out how so many axles get bent.