if a clean won't work, try completely removing the derived data folder.
the default location is in
/Users/you/Library/Developer/Xcode/DerivedData . if you haven't played with the DerivedData location in preferences, this is likely where you'll find the sub-folder containing the cache.
when i encounter a situation similar to the original question (retaining items i've deleted, or similarly missing items i've added or holding onto project icons i've changed), i perform a clean on my project, close it, hit Delete… in the organizer, possibly even remove it from the organizer, possibly even remove from disk and then re-checkout from git if you have it under version control in this way, then re-open the project from scratch.