Generally speaking you'd just charge a fee for the work. Royalties are mainly used for things like merchandise or other works based on your original creations, i.e. anything that is basically a re-selling of your work in a new format (mugs, t-shirts, pin badges, etc).
In this instance (and in most others) I'd just insist on an upfront fee for the work and retention of copyrights, with an agreement in place that your work is not to be sold without a new agreement covering royalties. If the author wants to sell additional merchandise using your work, then I'd charge royaties.
Most self-published books won't sell anywhere near enough for a royalties option to ever be viable, and many people will try and get a smaller fee for work by promising future royalties, say 10% of each sale. A good equation to remember is 10% of nothing is nothing, so make sure you get paid in full for all the work you do and don't gamble on getting paid properly further down the line. I can almost guarantee you won't be.