"...so that I have to declare it every time I visit the UK?..."
Strictly speaking no, you do not have to declare a prior refusal every time you visit. Every time you visit you will have to fill out a landing card, and there's no questions on the landing card about prior refusals.
"...where it says "have you ever been declined a visa?"...'
On the other hand, you will have to declare a prior refusal to receive an EEA Family Permit because they ask for it in Section 6 of the application.
6.3 Have you ever been refused a visa for any country, including the UK? Put a cross (x) in the relevant box
The permit allows for multiple visits over the course of 6 months. Once you receive the permit, questions about your history are finished until the permit expires. Even if you visit 100+ times!
So you will only need to declare it once every 6 months regardless of how many times you visit. You can reduce it further be getting a UK Residence Card. When, and if, you ultimately qualify for permanent residence, you will not need visas any more and hence no need for family permit applications.
You can see the current landing cards at https://www.google.com/search?q=uk+landing+card&es_sm=122&tbm=isch&tbo=u&source=univ&sa=X&ei=ma1WVeKTG-Oz7gbq9oAo&ved=0CB0QsAQ&biw=1444&bih=798
For your other question about whether to appeal or submit a fresh application... I would submit a fresh application because I doubt an appeal would even reach the hearing stage. If your passport was not submitted, the Home Office have a defensible position and will tell the Tribunal that a fresh application can be made free of charge and you are just trying to 'clear your name' by introducing new evidence that was not available when they made the decision. And if the judge overrules and wants to schedule a hearing anyway, the Home Office will fight it every step of the way and further appeal any adverse decisions. All of that will take a long time, and you may not even win.