The handsome actor told Ellen what he and his wife feel about being told he wouldn't ever actually be in a relationship with Lena Dunham.