It may be better to get a used VHF/UHF dual band mobile radio and a used 12v power supply as well as a base antenna (or even a dual band mag mount stuck on a steal cookie sheet) than the typical handheld radio as your starter. If your area has little UHF activity, you can save even more with a used VHF only radio. Just make sure that the radio will allow you to program the generally needed PL (or CTCSS) tones so you can use most repeaters that require them.
While handhelds are all-in-one, they often don't work all that well due to the low power, poor antenna, and always moving polarity. You'll probably hear others talking but have difficulty talking to them or get the very common indication of a handheld response, which is "Sorry, you're not making it. Try to increase your power or improve your location.". A handheld does make a good second radio, but as your first radio or as a primary radio, not so much.
If you have some "skills", you can save on the antenna by building your own (there are many designs so I'll let you Google them for yourself). You may also have some of the other parts already on hand if your junk pile is well stocked. I also recommend you check out any local ham radio clubs and go to a meeting. They'll generally be happy to let you visit and often will be quite helpful in assisting a new ham get started in the hobby.
Also if there's a hamfest (basically a ham radio show with vendors, swap tables (aka used equipment marketplaces), and/or presentations of interest to hams. With help from a member of that club you visited you may find a complete starter station for minimal cash outlay