Sometimes you remind me a lot of James. He called it my 'furry little problem' in company. Many people were under the impression that I owned a badly behaved rabbit.

The quote means that someone else (James) used to joke about the speaker owning a badly behaved rabbit in the presence of others. The phrase “furry little problem” was the phrase that James used to tease the speaker about owning a misbehaved rabbit. This phrase created the impression that the speaker indeed owned a rabbit, but the speaker never clarified, leading to confusion among those who heard the conversation.