Well I havent set out to duel anyone yet, just defending myself when challenged, just sux you dont have a choice... I just got challenged again while is was in the Barbarian village fighting a barbarian (won me a pair of shoes too)
I just keep spending my gold so i dont have much for them to take if they beat me.
I'm up to level 3, Armor 97, broad sword 3-5, 12-12-11-10-10. Work the stables a few hours, spend it on training, go to the tavern do a quest, head for mountains or forest, battle wolfs or harpys... earn gold, go back and upgrade weapons and armor, rinse repeat.
I think Charisma must relate to your chance to hit % I uped my charisma from 8 to 10 and my chance to hit % went from 48% to 60%