Most folks will judge your macro-work without the aid of a micro-scope. IOW, is this sort of 'boo-boo' visible/obvious to the uninitiated standing erect at the side of the layout as the subject passes by at a scale 60 mph?? Probably not.
The exception will be if your handiwork goes before the squinty-eyed folks judging NMRA contest models.
Your work on the object-at-hand is Grade A, IMHO. You're being too fussy...IMHO Eliminating those nearly invisible defects is hardly a hill to die on when considering the overall, ever-growing project list before your lifelong participation in the hobby...IMHO, again.
OTOH... Some have commented using a plate of glass as a flat surface for sanding matching surfaces. I tried that...and promptly broke the plate of glass trying to safely store it away! Better idea, thought I: The flat steel surface of my table saw. I happen to have an industrial version of the same, with a machine ground cast metal flat table surface. For sanding, it works as well as glass. I also later acquired a scrap piece (about 12" square) of 1/2" thick Baltic birch plywood ( 7 plies of uniform thickness) from a cabinet carpenter doing a job for us. I also use that for a sanding surface, periodically checking it's lie on that table saw surface to be sure it's acquired no shipbuilder's sheer over time...which it hasn't.
Meanwhile, back to your own critical standards of workmanship. I once had similar angsts. (I also once had hair on the top of my head, too!) When I worked at the LHS after retiring from the corporate world, my fellow staff members would chide me for my own critique of my work. They convinced me that it was time to graduate to the "Looks-good-enough!" standard assessment.
From which I proudly acquired a nickname...and a more healthy, satisfying view of the hobby.
KD...a.k.a., Lucas Gudinov