$3 and $1 were jewelry pieces and not circulating coins after the Civil War. There are frequent archival letters from jewelers asking for these to use in bangles and pins. The others were around but letters suggest people found local and city banks did not have gold coins to pay out....especially bright, new coins. Depositing a good US gold coin in order to receive new coins in return was prohibited, but the Mints and Treasury routinely exchanged a few gold pieces for collectors or people wanting a new half eagle for a present. The volume of gold recoinage implies these denominations circulated, although mainly in the West and South coast ports. (The 1875 letter - above - from the Asst Treasurer, New Orleans, is illustrative.)
(Note that banks were required to hold reserves in various kind of debt instruments and gold coin. However, these reserves could not be paid out. It didn't matter that XYZ National Bank held $50,000 in gold coin as reserves, the coins could not be paid to a customer.)