Welcome to the forum and the hobby. I will start by echoing what many here and on the PCGS forum have said.
1) Starting out by buying a ton of "stuff" seems fun and in some ways it can be. There is a bit of magic in searching thru a lot of coins and filling folders, many older coin collectors started out that way. But what we found was that at some point we wanted to specialize and then we had to dump all that "stuff" (usually for very little money) and start again. I will not discourage you from doing some of that but I would encourage you to not go overboard.
2) Keep an eye out in the local paper or media for estate sales, this is one of the best places to find random lots of coins. Do take care as many do not know what they have and may be asking far more than the coins are worth.
3) Get a redbook, the pricing will be out of date very quickly but it will give you an idea of what dates and mintmarks are worth more than the very common ones and allow you to see if the coins you are looking to buy are reasonably price or not.
4) My last and in my opinion the most important point is to take your time and have fun, this is a hobby so enjoy it.