Use ctrl+shift+right click to remove it.

Or if you're using the Android version, you are out of luck (I didn't think that part through until now)