Lots of good advice here. I like to go to a show with a list of specific things I am looking for, but I will still be open to an impulse purchase of something that will work with my layout.
I walk through the show and look over every table, but will stop for something on my list and make an offer. If I don't see anything on my list, I'll walk through the whole show then go back a second and even third time to see if anything strikes my fancy.
I'm fine with driving 2 hours to a show (they are infrequent and far apart in SC) and coming home with nothing. I still enjoy looking at all of the trains, buildings, etc. After 20 years back in the hobby, I'm still amazed at the variations in condition and pricing from table to table. At many shows you can find the same item in the same condition in different places for prices 30% apart.
I think the most important thing is to know yourself. If you like running conventional, then you might want to think twice before plunking down big bucks for a gorgeous command locomotive and vice versa (buying a conventional loco because it is such a good deal even though you run 90% command).
Courtesy goes a long way on both sides of the table. Unless a price is outstanding, I'll offer less, but always in a polite way. It is one thing to say "I was hoping for this in a slightly better condition, there are some missing parts and some rust, but if you'll consider XX$, I'll fix it up and take it. It is another thing to say, "Look at it. The paint is chipped, the plastic in the numberboard is missing, there's rust on the back truck, I'll give you half what you are asking and you'll be lucky to get that."
The same is true on the other side of the table. If the seller is nasty, contrary, ignores me for a long time for no apparent reason, then, no purchase, period. I don't need any train that much. Similarly, when a seller has 3 people waiting that isn't the time for a customer to expect a long discussion on the merits of a particular item.
I go to train shows to enjoy the merchandise, enjoy the conversations with sellers, get questions answered (often if a vendor doesn't have what I'm looking for, he'll know where I can find it), and generally look at it as a pleasant way to spend a morning or afternoon. Coming home with more toys is a bonus!