There are some MVNO operators (which rent the antennas from one of the four networks, i.e. Orange, SFR, Bouygues and Free) which have interesting offers, usually much cheaper than the big-name brands, with the inconvenience of having no access to wireless hotspots and sometimes lack of phone support (usually you have to use their forums to get support).
I am currently with Joe Mobile, mainly because their offer allows me to have cheap pre-paid Internet, using only a debit card (they ask you for a document number, but as far as I know you do not need to send a scan). 1€ for the SIM card (sent by mail) + 5€/month for 2 hours of voice calls (no special numbers) + unlimited SMS (+ 100 MB of data, which you won't use).
One of their main advantages (besides no contract) is that you can quit very easily: just go to the website and click "Off". It will put your line in "pause" mode, so once you finish the current 30 day-period (already paid for), you do not pay anything more and your line becomes inactive in a few months. You can reactivate it later if you want, or just let it "die".
Also, you might consider benefiting from parrainage offers (invitations), where each additional phone line means a discount for the inviter, for the invited, or both. These offers may work even if you have never met the other person, e.g. via the forum. Just go to one of the MVNO provider's forums, you can often see people spamming invitations (which will benefit both of you).
One liability of Joe Mobile is the SFR network, which is quite bad where I live, but it used to be fine where I lived before.
Edit: I have just been informed by Joe that they will stop operating on June 15th, 2015 (acquired by Virgin Mobile).
There are several such low-cost providers nowadays, but their names are harder to Google if you don't know them. Some French websites offer comparisons between them, but pay attention to some details such as cancellation conditions and roaming mode (some of them require it to be activated before leaving France).