According to the instructions for Form DS-11 (U.S. Passport Application):
Section 6039E of the Internal Revenue Code (26 U.S.C. 6039E) and 22 U.S.C 2714a(f) require you to provide your Social Security number (SSN), if you have one, when you apply for or renew a U.S. passport. If you have never been issued a SSN, you must enter zeros in box #5 of this form. If you are residing abroad, you must also provide the name of the foreign country in which you are residing. The U.S. Department of State must provide your SSN and foreign residence information to the U.S. Department of the Treasury. If you fail to provide the information, your application may be denied and you are subject to a $500 penalty enforced by the IRS. All questions on this matter should be referred to the nearest IRS office.
Other sections of the instructions say that if you fail to provide your social security number, then your application "MAY RESULT IN SIGNIFICANT PROCESSING DELAYS AND/OR THE DENIAL OF YOUR APPLICATION". However, if you are sure that you have never had an SSN, then the above instructions to fill in zeros apply.