When Sam woke up with his soul and hugged Dean and Bobby
Cas must have been in the room, hidden and watching
and when he visited Sam he went for a hug cos he thought it’s natural, for someone who came back from the dead to hug those who cared greatly about him
and Sam just rejected him
but Cas didn’t know it’s because Sammy’s pure conscience wouldn’t let him hug the person he was about to use for information
he just thought ”Oh okay I guess I’m not as important to you as Dean and Bobby… Even though I did everything in my power to save you and worried about your soul just as much as them…”