This quote means that the speaker understands that not everyone gets their sense of humor and they find it hard to understand why some people do get it. However, when someone does get the speaker’s sense of humor the speaker will try to keep them close to them, as these people are valuable to them.