Project #1Ok, after thinking about it for a while, i realize that it's not possible to find the dmg formula, because there are some unknown factors involved...
When u fight a player lvl 80 and a mob lvl 80, u notice that u deal almost the same dmg (if they have the same defense). Although, a player lvl 80 has at least 70% dmg absorption in accessories. So i think that mob have absorption too, but the value is unknown. So there is no need to continue in searching.
Project #2I believe there is no best build dmg to fit with all type of enemies. Because the enemies' defense change depend on their distribution of stat ie a pure str garment have 1000 mag def less than a pure int in garment. Because we are talking about hybrid, so what we interest is the magical dmg part. A pure int has very high mag def, so a little bonus hybriding mag dmg is nothing to him. So to fight him, just build a pure str, and kill it with crit.
Again, this is just my thought, D2A can say anything what he wants, but telling the opinion and the bullshit is different

What we interest now is a str hybrid with tons of HP but deal good dmg to other str build. Although, there are a lot of weak hybrid deal less dmg than a str, i dont know why... maybe because the mag dmg is not high enough to overtake the defense.
To make it simple, i'll just consider a simple dmg formula:
dmg = (PhyPower * PhyBalance - PhyDef) + (MagPower * MagBalance - MagDef)
We interest in mag dmg part. If u found a negative resultt, consider it a reduction of dmg. So the best dmg build will be a build that has none reduction. The mag power must be higher than mag defense to have a positive value. Dmg also depends on what weapon u have. If u think u have a sosun+9 crit 20, there is no need to go hybrid
I made a table of value, coz it's too big to post SS, u can DL it here
http://www.mediafire.com/?1gd9gtcjygcStats are based on cap 100, ksro data. Exel file also include dmg vs (str+shield), pure int, (int+shield); stat after add blues, etc...
U can modify values to test with other build if u want. If u dont have exel, u can read with notepad, but to modify, ur farked.

vs int+protector balance 88:42
vs int+protector+shield balance 83:49
after add blues gear +66str/int
vs str+garment can be pure str
vs str+garment+shield balance 97:29 (for bow)
vs int+protector like above
vs int+protector+shield balance 92:36
So in the end, the build that can work out with most case is 92:36, for bow
Fighting a pure int garment, just kill him with pure str crit, dont try to build a hybrid str that work with him, coz u'll sacrify a lot of HP ...