Further, he didn't see them on the way back to the Hogwarts Express during his fifth year because he didn't fully appreciate Cedric’s death. Thus, he couldn’t see them until HBP.
Wait - didn't he see them fifth year? (Cedric was killed in the fourth year) That's the discussion going on with the current movie, too (OotP)
Threaded Mode