I have always wondered about that - and perhaps it is because they never considered him "one of them". Awful if that is the case.
Of course the problem often is that adopted children are loved very much by their adopted parents but other relatives just cannot understand why people adopt.
There are plenty of childless people who would never consider adoption as an option because they cannot comprehend the concept. I think there are more males than females who feel they only want children with their own genes. So it's hardly surprising that relatives, who have produced their own children and have never suffered the emptiness of childlessness and maybe lack imagination, may not whole heartedly embrace an outsider as the next heir to the throne.
I can imagine them watching askance as Jeremy and Sheila, both undeniably good looking and glamorous seeming to do exactly what they wanted, taking money from their parents left, right and centre. I would argue that although there were personality clashes Sheila and Jeremy were in many ways doted on by their parents.
It was certainly a very different situation than the genetic children of Robert and Pamela Boutflour who were expected to toe the line and do as they were told at all times by their dominating, controlling father.
I'm quite sure that branch of the family just didn't understand the Bambers and their children ie. the reasons for having them and the reasons for allowing them to live such chaotic lives.
From this perspective, Sheila and Jeremy would never have been 'one of us' . imo