I cannot remove a product from my selection?

Short answer: The product is still presented in one of your shops. Take it out of the collection, put the draft into service, and it then becomes removable from the selection.

Cause

The selection is the reserve your shops draw from. A product still presented somewhere therefore cannot leave it: removing it would leave a shop with a product that has no source.

The “Remove this product from my selection” button is then grayed out, rather than active and then refused. Hovering over it tells you why, naming the collections that use the product.

It is your live shops that count, not your drafts

Taking the product out of a draft's collection is not enough: as long as the draft has not been put into service, the live shop still presents the product — and the button stays grayed out.

The block disappears when the product really is presented nowhere, so after it has been put into service.

Solution

Open a draft of the shop concerned — a live shop cannot be modified directly (orange “Modify the shop (create a draft)” button).
In the collection, remove the product.
Click “Put this draft shop into service” (blue button with two checkmarks). This is the step that actually frees the product.
Repeat for every shop named by the tooltip — it names them all, and a single one forgotten is enough to keep the block.
Go back to the selection: the “Remove this product from my selection” button is now active.

If the product exists in several versions, a checkbox decides whether the action also applies to the other versions. On a product already selected, it reads “Apply the below action also to the other versions of this product” — check it to remove every version at once, uncheck it to act on this one only. See What is a multi-version product.

Should you really remove it

Rarely. A product in the selection costs nothing and is not visible to buyers as long as it is not in a collection — see Catalog, selection, collection — what is the difference.

Leaving it in reserve lets you put it back into a collection later with one drag and drop, without going through the catalog again or redoing its colors. If your only goal is for it to disappear from the shop, take it out of the collection and stop there.