I've ridden the Auto Train, Palmetto and Silver Meteor, as well as commuting on the Long Island Railroad for a summer. I enjoy riding the train, but it is slow and the cleanliness of the cars on the LIRR could be much better.
Human labor is expensive, but a key issue is that the rail infrastructure we use is, in many cases, over a hundred years old or at best several decades old. There has never been a national effort to upgrade our rail network and social changes have made rebuilding/expanding it much more costly and difficult. Adding a second track or third track requires environmental impact statements, studies, fighting through NIMBY lawsuits and lengthy and expensive land condemnation proceedings, etc. Add the high cost of labor and the cost of building a 21st century rail system with multiple high speed tracks between major cities soars beyond what taxpayers are willing to fund or private rail corporations would find profitable.
The days when you could pay thousands of immigrants minimal wages to lay rail (or build roads for that matter) are over. Today we see much smaller crews using very expensive machinery to do the jobs. The work proceeds much more slowly and at a huge financial cost.
We are stuck with an aged infrastructure while our economic competitors have much newer and more cost effective and faster infrastructure. Much of Europe was rebuilt after WWII (with help from Marshall Plan $) and much of Asian infrastructure is very new. We are at a competitive disadvantage.
We can argue the merits of not updating our infrastructure in favor of spending tax dollars on social spending programs, but the current situation is what it is. The cost of the needed upgrades is well beyond what the class I RR's can afford and unless there is a tremendous national effort to provide regulatory relief and trillions in funding, change will be incremental and passenger rail transportation will not achieve anywhere near its potential. And, that's unfortunate.
I'd much rather take a fast train from SC to NYC than drive I-95, but that just isn't a realistic option in most cases.