Ex husband saying I was abusive... will I lose custody of my daughter?

Courts lean toward women no matter what anyone says. Unless your husband called the police at the time, claiming spousal abuse will be difficult. He would have to prove it. (Normally, women just have to claim it.) While you both have issues, the children have to go somewhere. I suspect you'll get them. Besides, he doesn't really want them, it would interfere with his drinking and bedding women. However, you claiming you don't want him around your daughter until he gets his life together is hypocritical. You "ain't" Mother Teresa, you know! Let this play out naturally. He is going to get hit with child support payments, which he probably will get behind. You then will be able to get him arrested and sent to jail until he makes child support payments. Before this is over, you will get your "pound of flesh". Relax. You have the upper hand.