This behavior is correct as the Member property of a group is modified.
As a workaround, you can add the If PowerShell script returns true condition to the Business Rule with the following script:
$Context.ConditionIsMet = $True
$operation = $Context.Action.GetOperationName($NULL, $Context.TargetObject, " ADM_ACTIONNAMEFORMAT_GENERAL")
If ($operation -eq "manage group members")
$Context.ConditionIsMet = $False
The script returns False if only Member property is updated, and the Business Rule is not triggered.