I use a PAINT brush (about 1/2 inch wide) to brush out the ground coffee, then use a damp paper towel to clean up the fine powder. This method works very well, especially if you use a soft brush. Keep the brush near the grinder and brush out the grinder after each use.
The bottom of the bowl where the blade is, is metal for about 1/2" to 3/4" in the deepest part in the center. Above that it is white plastic and the smoky colored plastic cap fits down into this area while grinding. My new grinder has no chipping, and my old Krups grinder, (same design, purchased and used since the mid '80's!) has only one tiny bit of chipping on the very edge of the plastic cap. But it was used daily grinding coffee beans. My old grinder has held up over those 30 years better than I have. LOL! The cap on my old grinder did not cloud up over time.
I keep the grinder for coffee for coffee alone. If we are talking about spices, there is often a sort of co-mingling in the course of a dish so it isn't that important if there is a little mix in the flavorings. I do wish it very carefully with detergent and water but would not guarrantee no cross-contamination.