This phrase is expressing the idea of emotion of fear and a desire to stay together. The speaker is saying that, if they thought the other person would forget them after they leave, it would make it too difficult for them to go. The speaker is asking for a promise that the other person will not forget them, and in this way they can have the comfort they need to leave.