Should I let him make the mistake of uming I've had a promiscuous past?

I'm 25, haven't had many relationships, but I’ve never cheated on anyone. My current guy is 30, has never had a serious relationship, & has had fewer partners than me. He’s the 2nd person I’ve loved, the 1st person I’ve felt this close to. This past month, he’s been on the other side of the country caring for his dying father. We talk on the phone 2 hrs/day; I’m faithful to him, though there’s no promise he’ll be back at all. I’m on the West Coast for grad school & he is taking some time off his post-doc to deal with this. He loves me, but recently said he’s having 2nd thoughts as to whether I am “the 1” because he thinks I’ve had a promiscuous past. I haven’t, but it hurts me that he cares, & I don’t feel like defending myself. MY love is unconditional. I would like to know that he loves me for who I am despite his (unfounded) fears about my past. I’ve been honest with him, but he says it hurts that I don’t fight harder to convince him that he’s wrong. I could; but should I bother?