I think that's what made audiences connect with him is that he's closer to the everyman, not just as in a very average physique, but that he gets scared, makes risky decisions, and uses his dry humor to comment to keep his sanity together.
I and many others find it refreshing to have an extraordinary hero without the aura of invincibility.
AMEN, Robert! Yes, that is what made the original Die Hard so great: the hero was REAL, and not superficial. The picture couldn't have succeeded with a "big guy" trying to imitate Arnold Schwarzenegger and Sylvester Stallone.
You do realize that what you are complaining about is one of the things that made the original Die Hard stand out from other action movies? No, McClane isn't a mountain of muscle like Arnold never-can-spell-his-last-name-correctly, a professional martial artist like Chuck Norris or Steven Segal, or a highly trained killing machine like Rambo or James Bond, but he isn't meant to be any of those. He's just a regular cop in the wrong place at the wrong time; he gets hurt, he isn't immune to fear, he cries.
You are correct that Willis wasn't a huge muscle man but if Arnold or some other pro weight lifter had been cast instead then Die Hard would have most likely turned out to be just another Rambo or Terminator-esque movie with a nigh invulnerable superman lead rather than an everyman character who is scared he might be killed at any second.