It's probably because you're wearing an armor, wich have weight.
Every unit of weight put malus on your stats, mainly Agi and Will. So if an armor weight 2, you'll have a -2 malus to those.
You'd have to have encumbrance skill, accessed by leveling up fortitude, to negate the effect.
Having less dodge open you to more critical hit, wich is bad, and will help removing negative effect. TLDR, dont put armor if you cant handle them in most of the case. Else, there would only be mage wearing heavy plates.