At the age of 61,Neville had done a full day harvesting,still working into the night after his supper. I would have doubted very much if he'd been up to struggling with anyone by the time he'd returned from taking the dogs out even later.
He might have been 6' 4,but he wasn't superman. Plus,as BW had remarked,he wasn't looking well.