Exactly. If you don't talk to me/keep in contact, I will assume you don't care or think of me which is fine. Just don't say I never bothered. You went away not me. 

I don't forget anyone.