I highlighted a possible reason for this. It may stemmed from the relationship he had with Sheila and may have been about good child/bad child and sibling rivalry. Sheila was almost 3 years Jeremy's senior and had received the lion's share of attention until Jeremy's arrival when she had to share it which she may have resented.
Did Jeremy learn, at an early age, that Sheila's needs came first? Were Sheila's demands for attention put before Jeremy's needs. Were the Bambers a little afraid of their volatile daughter and thus go out of their way to appease her. Did Jeremy feel it was useless to tell his parents how he felt because he saw that it took all their time to cope with Sheila? We know that Neville had lengthy phone calls with Sheila during which he calmed her down. Might she have thrown tantrums as a child to deflect attention from Jeremy? We know that Sheila changed schools twice and on occasion ran away. There is far more about Sheila, than Jeremy, to suggest a difficult and troubled childhood. Did Jeremy suppress his own feelings to the point where, seeing his parents yet AGAIN, turning themselves inside out to try to solve Sheila's problems -maybe at his expense- they exploded?
I don't really see it like that because Jeremy seemed to be coping until eight years old when he was sent away,and afterwards he would only see Sheila in the holidays,when they were left to their own devices and starved of the affection they needed. Jeremy would tag onto Sheila's clique in those early days before her illness,yet later she would become an embarrassment to him,and the perceived inequality would rankle as he put in yet another long week at the Farm whilst his sister lived in the Maida Vale flat. However we don't know the full extent of Jeremy's thought processes,only getting a smattering through Julie's statement,where Sheila is supposed to have "done horrid things" to him in the past,but when pressed for more detail he has never elucidated.