Do some reading and studying first.
I know it is tempting to jump in and start buying anything and everything, but, without educating yourself on circulated vs mint state, cleaned vs original surfaces, counterfeit vs real, you may be setting yourself up for some serious disappointment. That is not to say that you couldn't go on Ebay, bid on a few nice looking coins, and come out a winner, but, the odds will be more in your favor if you know what to look for first.
That said, raw BU Lincolns, Jeffersons and Roosevelts can still readily be had for a few bucks, and plenty of nice circulated older coins are still inexpensive and plentiful. Buffalo nickels and Indian cents are two that come to mind in that category. Of course, you will have to spend more for the scarcer dates, but you can get many of the more common ones in some pretty nice condition for not a whole lot of money.
And, as Coinbuf mentioned, there are hundreds of world coins that are available for cheap. In fact, many dealers at coin shops and shows will have a bin, bowl, or bucket of world coins that are marked 2, 3 or 4 for a dollar.
Main thing: enjoy what you collect, and collect what you enjoy.