Although an answer has already been selected, I don't feel either of the answers is fully satisfactory.
Quite simply, you only need a TV license if you are actually watching or recording live TV from any broadcast television station, either via terrestrial broadcast, cable, satellite, or over the internet.
That means you don't need a license to watch Youtube/Netflix (not broadcast TV, only available online). You don't need a license to use online catch-up services. You don't need a license to own equipment capable of receiving such TV, only the actual act of receiving the signal to either watch or record requires a license.
Also, there is no obligation to tell TV licensing that you don't need a license, even though they will tell you the burden is on you. As with all criminal trials in the UK, the burden is on the prosecuting party (in this case the TV Licensing Authority) to prove beyond reasonable doubt in a court of law that you were illegally receiving a TV signal. You do not need to let agents of TV licensing into your house, unless they have a search warrant. They are not police officers and have no police powers, they are employees of a private company.
Basically as long as you have a TV, even disconnected in your home, you must pay TV license.
So NOT to pay TV License from JAN 2015 you will need to meet the following criteria.
No TV in the house.
No ADSL/DSL capable phone line in the house.
Completely incorrect.
In answer to your question, if "very rarely 'watch TV'" means you ever watch or record live TV, as it is broadcasted, by any method, you need a license at the time you are watching/recording.
Sources:
http://www.lime-marmalade.net/faq.html#injunction
http://www.moneysavingexpert.com/utilities/tv-licence