Yes, yes and no.
Penalties are unknown.
You are strongly advised to use a tax advisor (Steuerberater), since you don't seem to know how these things work.
Summery from Blog listed below (in German).
Do I have to inform the Finanzamt immediately?
Yes and this can be done online by filling the needed form
After registering with the tax office, you will receive your tax number and your VAT identification number (UID) if you have applied for one.
- the tax number will be needed for every bill
- the VAT identification number is needed when your are sending bills to other EU countries
Note: Here the Blog contridicts itsself by later stating outside the EU, but that is not correct based on the Wikipedia article.
You will also receive a payment plan from the tax office, on which the amount of the income tax to be paid is noted. This is usually payable every three months. Once a year you have to hand in the income tax return - as a small business owner a simple input-expense bill.
Same question for Health insurance.
Yes, since in Germany health insurance is mandatory, you must report the change of employer to them to remain insured
- in this case you are your own employer, so you must do this yourself
- just as you must pay the employers portion of the contribution as well as your own
Having a health insurance is also a requirement of the residence permit.
At the moment I have the public TK.
Privat health insurance is possible, but this is something that should be thought out very carefully
- in the long term it may become very expensive
- reverting back to a public healthcare is not possible
Tax advisor (Steuerberater)
Using a tax advisor to deal with these affairs (which in most cases was delt with previously by the employers) is often advised.
It also leaves you free to consentrate on things that are important to you, which is rarely what the tax office thinks is important
- and that is when the horror begins
The peace of mind alone is often worth the fee.