I think you'll find that there are laws against tearing down Grade "listeds". WHF was the property of a charity who might have had views about it's destruction.
Thinking about it, I probably wouldn't have wanted to move into it immediately, but Ann didn't which MIGHT suggest that some thought was put into it.
I believe it would be wrong to demolish such an historic building as WHF, it would seem like a final act of violence to me.
At the same time I can sympathise with the lack of understanding many have for the behaviour of AE and family which apart from any other reason does seem to suggest the family had coveted the house for themselves.
For all that I think we have to accept that we are all different and we all grieve in our own ways, what is comforting for some is 'creepy' for others and until we have walked in someone else's shoes we can't really judge.