This quote means that some people think the speaker is not very smart, but they hope that most people think that the speaker is very skilled at whatever it is that they do. In other words, despite some negative opinions, the speaker wishes that their abilities and work are seen in a positive light by the majority of people.