Why exactly would that be disturbing for Jeremy?
I think there were concerns with both Sheila and Jeremy,as Nevill was Sheila's parent of choice with the telephone calls whereas Jeremy tended to stick with June,from the Jubilee photograph in which we see a Jeremy having returned from Gresham's without qualifications and looking rather insecure. Maybe as Jeremy started to have a social life I think they drifted apart again:this would tie in with the Roger Wilkes book where it's stated problems arose in their relationship around 1978. With Sheila's illness followed by June's it's evident to me that Jeremy became unsettled again,not wishing to pursue an academic education to A Level but deep down not having any bonds with farm work,at least not from the bottom up which was Nevill's viewpoint on how his son's career should progress. We therefore have a tumbleweed Jeremy at this stage of his life,without the regular routine and discipline of Gresham's,which Jeremy came to regard as a complete waste of time and never forgave his parents for sending him to.
If we're ever talking about attachment disorders look no futher than Jeremy having been sent away at eight years old to this institution which was so alien to him,at a moment when he might just have begun to bond with his parents.