Would you go to bday party if THEIR kids were sick?

We had a b-day/Easter party event going on for our daughters cousin. She is turning 2. The mom called me and said that her girls (ages 2 and 5) had been running a fever and coughing really bad, and even she was a little congested now, but she broke their fevers with tylenol and the only reason she is going to continue the party is because the girls aren't acting miserable at all, they are running around, just coughing a lot. She said she understood if we didn't come, but I feel really bad saying we don't want to go, but to be honest, I really don't want to bring my kids (ages 3 and 9 months) somewhere where I know the little ones are sick. What would you do? How would you say you didn't want to go? Thanks