Nah, not really. In fact this is quite easy, but probably too fiddly for D&D 5e.
Take this as a though exercise.
I wouldn't use as a replacement for the actual rules, but maybe as an option for some players. Why not? If you do the math, you'll see they are getting +1, +2 damage at most...
But I digress.
Here is the deal: we are not messing with "to hit" bonus for now. Just weapon damage.
As you know, in modern D&D, your damage bonus is based on your Strength (or Dexterity, but let's start with Strength): subtract ten, divide by two, round down, and add to damage. So a 1d4 weapon in the hands of a Strength 16 or 17 fighter deals 1d4+3 damage.
I suggested some weapons should get twice this bonus in GURPS D&D, but we'll leave that aside for now.
My suggestion is something less extreme, although a bit crunchier: instead of subtract ten from ability and dividing by two, some weapons will divide your bonus by three or four, or even multiply it by 2/3 or 3/4.
Dark Souls calls this "scaling":
The Parameter Bonus rating, also known as Scaling, is a gameplay mechanic in Dark Souls. It indicates the level of bonus damage one can do with a weapon, based on the associated stats. This rating can be S, A, B, C, D or E (in order from most to least bonus for the associated skill). Strength and Dexterity will increase the physical damage...
So, an "Strength A" weapon will benefit greatly from Strength, while a "Strength E" weapon will barely benefit from it.
In this regard, Dark Souls is more interesting than 5e: depending on your "build", you might favor a greataxe over a greatsword, for example, something that is near impossible for the 5e fighter (although a greataxe might be slightly better for a barbarian).
Let's play with the concept.
We will use the letter "C" for the usual bonus to damage (i.e., Strength-10/2), and add some new possiblites:
Here is a nice table showing the exact numbers, up until that Barbarian with Strength 24:
Notice that the "C" column is the "ability modifier" you're used to.
So, let's get some of the worst weapons from 5e: the mace, the clubs, flail, morningstar, war pick and trident. Give them a "B" rating and, voilà: now the barbarian will pick the morningstar over the rapier!
Add great to the list... and now a Strength 20 fighter will pick the greataxe over the greatsword!
But wait - you might say - I actually LIKE my barbarians using rapiers, and I WANT the greatsword to be better than the greataxe!
Well, here is the deal: you can add Dexterity scaling to the weapon in addition to Strength scaling!
Say, for example, that you add "Strength C / Dexterity E" for all finesse weapons, all swords, and spears. The "finesse" trait just mean you can switch the two around (Dexterity C / Strength E).
Which means: the Strength 16 warrior will usually prefer a morning-star over a rapier. BUT if his Dexterity is 13, both weapons will have equal damage. If you have BOTH Strength 16 AND Dexterity 16, the rapier (or spear) is definitely better. And so on.
Of course, at this point you might be picking up you calculator...
Some perspective on damage
If you haven't read this, take a look.
In short: the +1 or +2 bonuses to damage these rules will give you are a lot smaller than they look, and only barbarians will get a few +3s.
Also, keep in mind that in most cases 1d6+4 is worse than 1d8+3 because of critical hits, specially for barbarians.
What's the point?
This solves a decent amount of problems:
- Does away with "useless" weapons.
- Creates more interesting, varied weapons,
- Strength is always useful, even with finesse weapons.
- Dexterity is useful for all fighters - even if you have high Strength.
- Odd abilities (11, 13, 15, 17, 19...) gain some utility.
- Encourages stronger characters to pick heavy weapons, and dexterous characters to pick lighter ones.
- More specifically, monks get additional reasons to use lighter weapons, barbarians get additional reasons to use heavier weapons, and fighters will be encouraged to use different weapons in accordance to their specific build,s most of which will be very flavorful.
My head hurts from the math! Let's see some actual examples!
Mace, Greatclub (Strength B)
There is little to no reason to use these weapons in 5e: the quarterstaff is versatile, in addition to being cheaper/lighter. With these rules, anyone with Strength 13+ will pick one of them over the quarterstaff.
Sickle (Strength C / Dexterity E)
You'd usually pick a handaxe over a sickle, but if you're dexterous enough, this weapon might be a better choice.
Greataxe (Strength B)
Once reserved to barbarians, now the greataxe is useful for anyone with Strength 13+. Because of how critical hits work, the greatsword has an almost identical damage, unless your Strength is REALLY high.
Greatsword (Strength C / Dexterity E)
Now the greatsword requires good Dexterity for maximum effect. Nice!
As you can see, I used only the letters B, C and E. You can easily add "Strength D/Dexterity D" to some weapons that require an equal amount of Strength and Dexterity, and even "Strength A" for GIANT weapons.
Maybe SIMPLE weapons would have Strength D as an option, to allow Strength 9 wizards to cause damage without a penalty.
Two weapon fighting? Maybe Strength letters get degraded by one level in the main hand, two levels in the off hand... Or something. The ideal is that Dexterity scaling allows dexterous characters to gain more benefits form dual wielding.
Ranged Weapons? Let us give them some Strength scaling, shall we? Or try this.
In Dark Souls, some weapons scale with "mental" abilities. A nice addition to spellcasters! Or check this out.
The 3/4, 2/3, 1/2, 1/3, 1/4 progression is arbitrary: you could easily use something less extreme (for example, 0.7, 0.6, 05., 0.4, 0.3).
Here is a smoother table if you want it. I think I actually prefer this one: