I agree with both of you, @Megamind & @luckyrat: Any variant of using the blacklists would undoubtedly mean intense maintenance.
Personally I would prefer an option to configure the buttons themselves.
Sidetrack :
That being said, I would also recommend to have two views for the options-dialogue:
1. The initial really short basic set of options, as is the case right now.
2. An additional wide ranging advanced set of options that has to be opened separately.
Then there is the actual option for the contextual buttons themselves. For me a simple on/off switch would do the trick, as I never really use the button-feature. But I feel that this would be counter-productive to the evolution of the software.
Maybe a selectable option to restrict the contextual buttons solely to the whitelist and not show them anywhere else would be a better approach?