As your question suggests, you should not have to pay social security tax in both countries, thanks to their totalization agreement. The US Social Security Administration (SSA) provides details on the US totalization agreements with each of 29 countries. Their page on the US totalization agreement with Japan answers your question as follows:
- If you are working in Japan, for a US employer who sent you to work in Japan for more than five years, then ... your Social Security coverage and taxes will be only with Japan.
- "To establish your exemption from coverage under the U.S. Social Security system, your employer in Japan must request a certificate of coverage (form J/USA 6) from the local Japanese social insurance agency that collects your Social Security taxes in Japan." The US SSA website gives detailed instructions for such a request. (Note that J/USA 6 is different from USA/J 6.)
- "Certificates of coverage issued by Japan should be retained by the employer in the United States in case of an audit by the Internal Revenue Service (IRS). No copies should be sent to IRS unless specifically requested by IRS."
The Japanese Social Security website also information about the totalization agreement, although what I found about your situation was still rather vague when I put it through Google translate:
As I understand it, this means your employer needs to submit social security taxes to Japan on your behalf, in order to get the certificate of coverage from Japan, so that they will not have to submit social security taxes to the US on your behalf. You said that you're already paying Japanese social security taxes, but you also said that your US employer is paying you "under a US address", so I must have misunderstood either the facts of your question or the mechanism by which Japan collects your social security taxes. If your Japanese social security taxes are being collected properly, then the Japanese Social Security office should be able to answer your question and give your employer the required form. With that form in hand and paying the Japanese social security taxes, your US employer should no longer pay or withhold US social security taxes for you.