I agree that it would have been very easy for Jeremy to come up with a lie about how he knew what was in the wallet, especially since the communication with Caroline was not direct, but by letter. He had plent of time to come up with something.
I just think he resented the question. It showed that Caroline had doubts about him, and he thought: none of her business, can´t be bothered.