I think, I may be wrong, but the only way round this is to just add fonts to Fontbook as and when you need them. It seems that once added, they’re always active and having 2000 fonts active is always going to have an effect.
But like I said, I may be wrong.