Note when he's affectionate.
If he's only affectionate with you when it's time to have sex, you know to let him go. The truth is that he fell out of love or like a long time ago, and now he's just sticking around for the free, frequent sex he knows you'll give him to get him to stay.
If he brushes off promises, you know to let him go.
End the relationship. Men who love/like/are interested in their woman will never, never break a promise without a really good explanation. Again, he's probably just in it for the sex.
If he doesn't bring you around his inner circle, that's when you know to let him go.
Men want to show off the top ladies in their life to their friends and family. If he's always over here with his friends and you're over there, stay over there. Tell him to hit the bricks.
If he used to do all the work when it came to getting in touch and setting up dates, and now he doesn't, it's time to let him go.
Love is a fickle thing, and doesn't come and go on a schedule. Make sure what he used to do he still does, or end the relationship.