British citizens are apparently immune from deportation from Ireland. Under Irish law, they have many of the rights afforded to Irish citizens. Since this state of affairs is not due to the countries' EU membership, it seems likely that the law would remain the same should the UK leave the EU.
As for the non-EU spouse, it seems likely that Ireland would impose the same rules that they now impose on the non-EU spouses of Irish citizens. They could also make up new and different rules, however; we can only guess. I do not know enough about the pre-EU laws to know whether they would offer any guidance about the likely approach.
Information about immigration for non-EU spouses of Irish citizens is available here: http://www.inis.gov.ie/en/INIS/Pages/WP07000024