Sure, you can do this. Just stay focused. By far, the biggest overall cost will be your reception (venue and catering). And the #1 way to keep costs down is limit your guest list. There are also extra costs associated with guests, such as postage, invites, centerpieces, favors if you want them, etc. So this is your big challenge.
I'd suggest registering at theknot.com or weddingwire and utilizing their services. You'll get a comprehensive list of all possible wedding costs (obviously, you won't use all of them). But it will ensure that you don't leave things off like license fee, gifts for bridesmaids, etc.
Chances are you won't be able to do a hotel venue where they do their own catering unless you keep it really small (like 30) and use a small side room. So start thinking of other options. Google your city and "weddings" "bring own catering" or something like this. There's always options that will pop up like museums, arboretums, university banquet rooms, etc. Start figuring costs. Then chances are you can find an inexpensive caterer for the food - barbecue, chinese, mexican, etc. To save costs, get an officiant and do ceremony/reception all at same place.
Also, the venue and caterer are the first things you need to get lined up. You don't say when your time frame is, but places get booked quickly on weekends, especially Saturdays. So this is your second task (the first being your guest list). On the list, I'd start with must haves and then branch "outward". You'll get a better idea what you can do once you start checking costs. If you wanted 100 and you end up realizing this will take $3500 of your budget, then obviously you need to slash at that number.
Also, obviously, lunch weddings are less expensive than dinner weddings. You may find yourself in the position of having to choose between inviting more people vs having that smaller evening event.