I worked in France from December 2012 until February 2016. I was obviously paying taxes during this period. In March 2016, I left to work in the United States and did nothing about my French tax situation. However, I just noticed that they are collecting 34 euros from my account each month now for almost a year although I haven't declared anything since I don't work there anymore. What should I do in this case?
Generally speaking, in France, any monthly payment is an advance payment, similar to the way withholding works in other country.
For income tax, I think you were still supposed to file tax returns this year (2017), for the income you received in 2016. Presumably, it would have taken care of any excess payment, which would have been refunded. Unfortunately, the window to submit your tax return for 2016 has already closed.
For local taxes (taxe d'habitation), you have to pay the whole year worth, based on where you lived on January 1st. It can work to your advantage or, in this case, mean that you have to pay the amount for a whole year even though you were only using that place for two months in 2016.
In any case, you are supposed to report relevant changes in your situation yourself. If you move from one place to another within France, there is a convenient online service to notify a number of agencies/departments at once but since you moved abroad, the only solution is to send a letter to the relevant tax office. You should definitely do that, it should at least stop the payments but I am not sure it will be enough to recover any money.
I don't know what the rules exactly are but, unlike other countries, the French tax office is exceedingly unlikely to go after you merely because you failed to report the address change in the timely manner.