Use Remove from Selection 
to remove the new selection from the currently selected objects.
To change the selection with Remove From Selection
-
With several objects selected, in the toolbar click Remove from Selection
. Click one of the selection tools and select objects within that selection. The objects you selected are removed from the selection.
-
Choose Edit > Select Mode > Remove from Selection.