This quote is basically saying that there are very few people whom the speaker really loves, and even fewer people of whom the speaker thinks well. Moreover, the more the speaker sees of the world, the more dissatisfied they become with it, and this further reinforces the idea that human characters are often inconsistent and it is difficult to rely on someone’s outward appearance of merit or sense.