To be more specific I want to forbid adding rule-based groups to specific group(s).
In this case, you need to use a business rule like the following:
In the condition, use the below script:
$Context.ConditionIsMet = $False
# Get member group type
if ("%adm-MemberObjectType%" -eq "group")
$member = $Context.BindToObject("Adaxes://%member%")
$memberGroupType = $member.MembershipType
$Context.ConditionIsMet = $memberGroupType -eq 1
But at the same time I need to forbid changing already added groups (even nested ones!) to rule-based group.
There is no need to do anything about that. It is not possible to add members to rule-based groups manually. They will only be added automatically based on the rules you specify.