This quote means that, when reflecting on something that could have been but never was, the most sorrowful realization is that it could have been, but never was. In other words, it is heartbreaking when we think back on the possibilities of what might have been, but never came to be.