I know next to nothing about French immigration law, but I have worked briefly at the Swedish Immigration Board. If you have a residence permit as a refugee in any of the Scandinavian countries, you effectively have a residence permit for any and all of the EU-countries, including France. Even if your residence permit is from a non-EU Scandinavian country (i.e. Norway or Iceland), the "Schengen" treaty will give you the right to reside in any EU country.
If you have not yet been issued a residence permit in the country where you are, you should wait until you have it before moving to France. You should also direct your question the (1.) the immigration authority with which you have filed for a refugee status residence permit and (2.) the French immigration authorities to get the latest and most accurate information.