Sam and Dean and Cas are all in a relationship. Sam knows that Dean and Cas have sex without him and do other things without him etc. when Sam goes to a bar to hang out with Cas and Dean he notices that everyone there is a couple and Dean/Cas is the couple, leaving Sam out. Sam thinks it's best to leave.