"I think the first time I realised Downton Abbey was a hit was when I was sitting in a tea shop in New York and the couple next to me were talking about Downton Abbey, and then they recognised me."