If my child gets into private school, do we have to pay the regular public school fees?

There are no fees for public school, just your tax dollars.  And yes, you still have to pay those even if your child is in private school.

I want to point out, however, that even if you have a child in private school, if your child has learning delays, your local Board of Education still has to provide and pay for services for your child if they are warranted.  I talk a lot about how to access these services in my book, Testing For Kindergarten.  Even when my son was in private nursery school, we got free services from the city for speech and OT support.  When there was no public school program appropriate to help my daughter with her non-verbal learning disability, we were able to get the city to help us pay for a special private school that she attended for kids with her type of learning issues.  These services are available across the country.