The question is actually also relevant for German catholics who for some reasons want to leave the church. My understanding is that in canon law and catholic theology, this is simply not possible. In German law, it definitely is, the German states do not endorse the Roman Catholic Church's view of who is a catholic or not and do not ask for any proof beyond your declaration (it's called – approximate translation - the “negative freedom of religion”).
You can therefore at any time decide to declare yourself Konfessionslos and stop paying the “church tax” by formally exiting the Church (Kirchenaustritt). As far as the Church is concerned, you would still be a catholic and you would also automatically be excommunicated. In practice, it means among other things that you could not marry in a catholic church and should not take part in the celebration of the Eucharist. None of this should be a concern if you are a non-believer.
There is one last type of action the Church can take against you if you work for a religious institution like a Catholic hospital or retirement home: Nurses have been fired for having left the Church and those terminations have been upheld by the courts. Wikipedia (in German) has details on all that.
Whether you already formally belong to the Church in Germany or are registering yourself in the country for the first time, it's really up to the Church to convince you to remain a member. In principle, local authorities have to accept your declaration without further proof or investigation.
However, there are some reports that telling the town official that you were baptized was enough for them to register you as a catholic (this is from a site critical of the whole system) or that people who registered themselves as catholic at some stage (without being aware of the consequences) got in trouble later on. Apparently, the people in question still wanted to actively contribute to the Church (in their country of origin) and didn't really consider themselves atheists. It's not clear whether you can really do that or if it's all or nothing (Wikipedia discusses a recent court case about this – note that in this case, it's the Church, or at least some part of it, that brought a complaint, not the tax office or the person in question).
So you do need to be careful to be absolutely clear when registering and if in doubt follow the procedure to formally “leave the church” (which typically involves submitting a declaration and paying a small fee). But the Church can't force you to remain registered as a catholic forever merely because you have been baptized. Beyond that, church finances is actually a provincial/state matter so the exact law will differ from Land to Land.