Fatigue is a powerful debuff that affects your hirdmen if you travel too far without making camp.
When plotting your path on the campaign map, keep an eye on the little black dots with Z's in them - each time your group passes such a dot, one or more of your hirdmen will become fatigued. Mouse-over the dot to see who's affected.
To get rid of Fatigue, make camp and ensure that each of your followers gets at least 2 shifts of Rest.
If a character has the Heavy Sleeper passive skill, they will only require 1 shift of Rest to get rid of Fatigue.