Why does everyone think being friends with a married man is playing with fire?

Because somewhere along the line feelings will develope, such as you are right now, and it might turn into a big mess, basically putting him in the middle. Think about his wife's feelings to, would you want your husband being best friends with another woman and sneaking to pay phones to call her and spend a lot of time with her? It would hurt you knowing that you husband is so close to another female, so there is nothing wrong with being friends, but don't let your feelings get the best of you, some day you will find that special someone and you will know how it feels.