No, officials have the authority to do so. They may need to take your passport to check for counterfeits. They are even able to seize your passport for a longer amount of time if you are the subject of a legal proceeding. In this case, a judge would issue a warrant allowing the government to keep your passport pending the outcome of the case.
However, under no circumstances should you give up your passport unless instructed to do so by an official from the consulate or embassy.
If they ask you for your passport, tell them that you don't have it with you but that you will bring it right away. If they question your ability to do so, tell them that you leave it at home sometimes and that you'll get it tomorrow when you go to work or school.
There has been some controversy over whether or not officers at U.S. borders can now simply take people's phones at will. The Department of Homeland Security claims that this is now allowed under an emergency provision of the Patriot Act. However, critics say that this goes beyond what was originally intended by Congress and that police should not be given this kind of power without a court order.
The last thing you want is for a border agent to start rummaging through your phone book looking for information about yourself or your family members.
The State Department has the authority, but not the obligation, to refuse to grant a passport to a person who has an outstanding arrest warrant for a felony. A federal court, a state court, a municipal authority, or a foreign government can issue this arrest warrant. The warrant must be from a country where the applicant is still free to leave.
A U.S. citizen may be denied a passport if there is reasonable grounds to believe that the citizen is involved in criminal activity. Conviction of a crime would certainly provide reason to deny a passport application. However, a crime conviction could also be based on an indictment (a charge filed by a prosecutor) or upon a plea agreement (an agreement by which someone pleads guilty). In such cases, the applicant will usually be given a hearing before a judge to determine whether there are reasonable grounds to deny the passport application. If so, the application will be denied and the case will probably go into civil court as part of an action by the United States to collect money damages against the convicted person.
An arrest warrant alone does not constitute reasonable grounds to deny an application for a passport. However, if the applicant has been arrested while aboard a flight bound for a foreign destination, the airline crew may decide to withhold boarding until the arrestee is brought down to the airport detention facility. If this happens, the applicant will likely not receive his or her passport until after they have departed the United States.