If your "residence card" says that it is the residence card of a family member of an EU citizen, then it is an "article 10" card, and you might be able to travel to Portugal without a visa. If it doesn't say that, then you will need a Schengen visa to go to Portugal.
To travel to Portugal without a visa, by virtue of an article 10 card, you must be traveling with your wife or traveling to join your wife. If your wife does not accompany you on this trip and your wife is not already in Portugal, you will need a Schengen visa to go to Portugal.
If you travel with your wife, or travel to join your wife, then Article 5, paragraph 2, of the free-movement directive 2004/38/EC will apply:
Right of entry
Without prejudice to the provisions on travel documents applicable to national border controls, Member States shall grant Union citizens leave to enter their territory with a valid identity card or passport and shall grant family members who are not nationals of a Member State leave to enter their territory with a valid passport.
No entry visa or equivalent formality may be imposed on Union citizens.
Family members who are not nationals of a Member State shall only be required to have an entry visa in accordance with Regulation (EC) No 539/2001 or, where appropriate, with national law. For the purposes of this Directive, possession of the valid residence card referred to in Article 10 shall exempt such family members from the visa requirement.
Member States shall grant such persons every facility to obtain the necessary visas. Such visas shall be issued free of charge as soon as possible and on the basis of an accelerated procedure.
The host Member State shall not place an entry or exit stamp in the passport of family members who are not nationals of a Member State provided that they present the residence card provided for in Article 10.