The layout is probably not worth much. Brass rail is no longer used much. Everyone wants nickel silver or steel rail. Brass tends to tarnish and loose conductivity on the surfaces. So the trackwork, beautiful as it might be, might not sell at all.
someone from the local O Scale (2-rail) club could give you some ideas on the the equipment. Or as you say, you may want to donate it to the that club if your father was a member. If you think about that, you might caveat it with they have to remove EVERYTHING from the basement and leave the place in good order. That way you might get the layout out, even if no one is interested.
Even though dealers only give you 1/3 or less "what it is worth", that is misleading. It is only worth that much to the dealer. They are going to be stuck with the same problem you have, how to sell it and how long it will take. They give you the money up front with no fuss and they may take years to move enough stuff to start to profit. Much of the older stuff has to be cleaned and refurbished for a dealer to sell as opposed to a private individual selling "as is". So the dealer has work to put into it, and in most states must pay personal property tax on his inventory which this becomes. So the price offer from a reputable dealer is not stealing or cheating you, it is realistic for him to stay in business.
