I recommend the try before you buy method, or at least only purchase games that have built in re-playability.
You've already got some good recommendations so far (although I think I wasted my money on GTA IV, I don't recommend purchasing it without playing it first), but some of those games lose their appeal after playing them through once. Bioshock is an example of this... it was an absolutely stunning play through, but I haven't picked it up once since completion. I'll definitely play it again sometime down the road, but not until I've forgotten exactly how the story goes, as unraveling the plot is crucial to the game's enjoyment.
However, a game like Call of Duty 4 can last you much longer, despite the single player experience not being quite as strong as in Bioshock because the built in multi-player can have you playing for weeks on end.
I think the best games to purchase (at full price) are:
COD4
Orange Box
Halo 3 (if you like it)
Oblivion
and whatever your favorite sports/racing games are
These games all guarantee lots and lots of play-time and will be constantly rotating in your 360 throughout its lifespan. You can probably add games like Fallout 3 and Fable 2 to that list and likely the Left 4 Dead and Gears of War 2 if those games have strong multiplayer. Games with open worlds/levels like Crackdown, GTA IV, Saints Row, Dead Rising, Bully, etc. can all last a long enough time if you like going back to them and can be worth a full purchase (I still play Crackdown every so often, what a ridiculous game).
Some games that I've enjoyed that I'd recommend renting and playing through are:
Bioshock
Mass Effect
Gears of War
Call of Duty 2
Rainbow Six: Vegas
Stranglehold
Devil May Cry 4
Ninja Gaiden 2
Mercenaries 2
These games can be beaten in a week or two if you get into them, but might not offer you any reason to keep them in your console after playing them from start to finish. Most action games are somewhere between 10-15 hours of gameplay, and don't have too much to do outside of the main quest. I rented all these, and the only one I purchased was COD2 because I saw it for $10 at the store. I'll probably buy Mass Effect when it's price drops and play it through again as well. However, I'm happy I only spent $5 renting these games and not $60 purchasing them. There are many upcoming action games in the fall that look promising (Dead Space, for sure) as well which will probably make for a fun renting season.
You also mentioned the possibility of buying arcade games. I've gotten a lot of value from Braid, Pac-Man CE, Castle Crashers, Geometry Wars 1 & 2, and Super Puzzle Fighter II Turbo (a family favorite). This nice thing with the arcade games is that you can try before you buy, which can save you money there as well.
There are plenty of other games that you might enjoy that didn't receive critical praise, but are nonetheless pretty fun like Assassin's Creed or Too Human (for example). I use Amazon.com as well and just fill up my "Wish List" with games that I hear good things about, then wait for them to fall to around $15 before I buy them. I just bought GRID (racing game) and I'm really enjoying it, I would definitely recommend you check out the demo for it and see it for yourself.
