You cannot really “overstay” in the way third-country citizens can. As a British citizen, the most that can happen if you stay longer than 3 months without doing anything in another EU country is the following:
- You might be asked to leave.
- You might have to pay a fine if you failed to complete some formalities.
The rest of the answer will explain how all this works but the most important thing is that you cannot be banned, detained or removed merely because you failed to complete some formalities in time or disregarded this three-month threshold.
The distinction is subtle but the three-month threshold does not quite work like a short-stay visa for non-EU citizens and this has important consequences for you. After three months, you need to qualify in one of four ways (more on that below) to have a right to stay under EU freedom of movement rules (that's completely unrelated to Schengen and also applies to other EU citizens in the UK, incidentally). But even if you don't, you are not accruing illegal presence or risking removal by staying in the country.
It's only if the authorities ask you to justify your right to stay than the rules become relevant. In countries like France and the UK, you can therefore basically stay indefinitely and fully legally as long as you don't attract attention to yourself or somehow make yourself undesirable. One way this could happen is if you apply for welfare benefits or free healthcare coverage. If you do not qualify for residence under freedom of movement rules, you are not entitled to welfare benefits so the authorities will usually ask you to show that you do when you apply for them and could even ask you to leave the country if you don't. But as long as you don't do that, nobody will check whether you qualify and you can stay for as long as you wish without trouble.
Denmark is a little different because, unlike France or the UK, they require EU citizens to register their presence and get a “certificate of residence” within three to six months of their entry into the country so it's more difficult to ‘fly under the radar’ and remain in the country without worrying about all this. At this point, since you have stayed more than three months, you are theoretically liable for a fine (I don't know about Denmark but in other countries I have heard about ~€500 fines so not completely insignificant).
The problem is that applying for this certificate of residence requires showing that you qualify for residence under EU freedom of movement rules so that might not be possible for you. And you would probably have to pay the fine anyway. But if you don't and are somehow found out, you could be asked to leave and would certainly be fined. And if you find a job, applying for the certificate and paying the fine will probably be necessary as well.
Whatever happens, you cannot be banned from Denmark merely for having stayed too long without registering properly. Even if you are asked/forced to leave the country, you could always come back for a short stay or, if you qualify, for a long stay. But escaping the fine could be difficult.
Now, I have used the phrase “qualify for EU freedom of movement” several times without clearly explaining what it means. You mentioned working and studying in your question and those are indeed two major ways to be covered by the EU freedom of movement. But there are a few others like being self-employed, living together with another EU citizen who qualifies or simply having sufficient financial resources to stay without becoming “a burden on the social assistance system” of the host state. So besides working or studying, you might still qualify if you show you have some savings or live with your British (or other EU citizen but not Danish) girlfriend/boyfriend/spouse. The page from the Statsforvaltningen I linked to above also details how you can qualify.
Note that – perhaps counterintuitively if you compare your situation to that of third-country nationals – working would help you, not hurt you. It's not a crime for you to work in Denmark and you don't need any permit or special qualification (in practice you probably need to register – and possibly pay a fine since you are now too late – to get a tax/social insurance number that your employer will require but those are just formalities). And once you work, you are covered by freedom of movement so you cannot be asked to leave unless there is a very serious reason (basically national security and the like).
In practice, if you now intend to work and settle in Denmark but you are concerned about the fine and not in a hurry, one thing you could do is leave for another EU country or go back to the UK for a few months and then return to Denmark later, ideally with a job lined up or perhaps to search for one intensively. You could then complete all formalities by the book starting from your new entry. Whether you would be fined for your previous stay/unregistered residence would depend on how much of a paper trail you have left and how diligently they check but my guess is that nobody will care about it.
