This quote is saying that when they were a child, they were not always the one who made funny jokes but they always knew who was funny and who they wanted to spend time with. They understood who had a good sense of humour, and they wanted to spend their time with those people.