I cherish the memory of being a friend of Frank Sinatra on a friendship level to the point where we really hung out. We worked in Vegas, we'd talk on the phone, and if I wasn't doing anything, I'd fly out, and I spent time in Palm Springs at his house - on a level the way friends would be, not with a whole crowd of people.

The speaker is talking about his friendship with Frank Sinatra. They were very close, to the point of spending time just the two of them together. They would work together in Las Vegas and talk over the phone. In addition, if the speaker was not doing anything he would fly out to see Sinatra, and even visited him at his house in Palm Springs. The relationship between them was very intimate and not just in a crowd of people.