You are asking about getting an entry clearance as a spouse, which is a settlement track and not a visitor track. The controlling reference for your question is Paragraph 28 of the Immigration Rules, which says...
An applicant for an entry clearance must be outside the United Kingdom
and Islands at the time of the application. An applicant for an entry
clearance who is seeking entry as a short-term student must apply to a
post designated by the Secretary of State to accept applications for
entry clearance for that purpose and from that category of applicant.
Subject to paragraph 28A, any other application must be made to the
post in the country or territory where the applicant is living which
has been designated by the Secretary of State to accept applications
for entry clearance for that purpose and from that category of
applicant. Where there is no such post the applicant must apply to the
appropriate designated post outside the country or territory where he
What this means for you is they expect you to apply for entry clearance as a spouse at the British High Commission in New Zealand. It is not necessary for your children to accompany you.
Additionally, there are some in-country routes such as the FLR FP route. This is not available if your leave to enter was issued for less than 6 months so if you try to apply they will see that you entered as a visitor and refuse the application (and keep the fee). Unless of course you can benefit from a corner case day counting method and out-whittle them, see a solicitor for that one.
As you are a parent of British citizens (or more appropriately, EEA citizens), you may also look at the Zambrano route, which also allows an in-country switch. The link takes you to Colin Yeo's site. He's a friend of mine and a widely acknowledged expert in Zambrano cases, so while not an 'official' source, the site is very accurate and reliable.
If your husband exercises treaty rights in the EEA for 6 months, the Singh route will be available to you.
Finally, you can brass it out and use the Human Rights Act to remain in the UK; the downside is that it takes horrendously long to achieve, introduces the stress that accompanies immigration limbo, and your settlement clock does not begin to tick until your leave to remain is granted.
Attempting any of the in-country routes (except Singh) without legal representation is perilous. Expect resistance and lengthy processing times.
Depending upon which route you ultimately select, your path to settlement in the UK will be either 5 or 10 or 20 years. Once you are granted settlement (ILR), you can go ahead and apply for naturalisation and a British passport. NOTE that when you originally arrived in the UK, you presented yourself as a visitor who intended to leave in six months and not as a person who intended to apply for settlement. It's unlikely that a person would take such an important step without being aware of it beforehand. They may review that topic again in more depth if you used an in-country route when you apply for naturalisation. Their conclusions may be adverse to your naturalisation application, but will not affect your permanent residence. They keep adding new hurdles to the nationality policies all the time; even now people who have relied upon concessionary provisions are being refused.
It's always best if you have doubts to get with a solicitor and ask more questions about each of these routes. You can print out this answer and show it to them if you need a reference point.