I think this is time to back down, say a quick 'sorry' and remove without further question. Lesson learnt, Always ask.
Building up a good relation with the client is very important. Whilst you need to remain professional, they are not your friend - lets make that clear or before you know it you'll be doing "quick favours". But happy customers are the key to successful freelancing. You never know who they might tell or when they might come back. I get the occasional long forgotten client crawl out of the wood work from many years ago due to them having a good experinece with me back then. Sometimes they've moved on and next thing you know you find yourself getting commissions from their new company.
On the other hand, to manage a problem client one technique that my colleague keeps going back to, and it seems to work is a full on toys out of the pram job - but in a very professional way.
Basically, whatever it costs you, however much time you've invested always be prepaired to stop work, offer a full refund and walk away. It's a game of call my bluff which if you're willing to carry out you will always have the upper hand.
"Yes, I'm very sorry, we seem to have reached a stale mate/have a difference of creative opinion .. etc etc. all I can do is offer a full refund. Obviosuly that means you won't have paid for any of the work I've done so far so I will retain copyright/IP etc... and I won't be able to provide any of the files, source code etc..."
Usually a client has a deadline and has already invested time in the selection process so the thought of having to find someone else to do the work at the 11th hour will cause them to back down! Just depends if you have the balls to see it through. Good luck.