I don't believe this is how colorblindness works. Colorblind people can still see colors, but certain colors they cannot differentiate. The most common type is red-green colorblindness, where those colors blend together.

To avoid this, UI components should be designed to be able to be differentiated. I would need opinions from people who are actually colorblind, if they find any part of our interface confusing or hard to read. We added a colorblind mode to an app I develop at work, the only change it makes is that one of the colors on the UI is changed slightly. An actual colorblind person suggested it.

As for the elements, we have elements spanning the entire color palette. There isn't much we can do to solve this problem. Though if there were some really common elements with issues I would also like to know.