index=_internal sourcetype=splunkd NOT info (log_level="WARN" OR log_level="ERROR" OR log_level="CRIT" OR log_level="FATAL") splunksh* component=SHCMasterHTTPProxy log_level=WARN
06-08-2021 17:11:11.735 -0400 WARN SHCMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/artifacts/dummy/remove_all captain=splunkshc-prod05.med.umich.edu:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line="Internal Server Error" transaction_error="<response>\n <messages>\n <msg type="ERROR">Peer 3AB94F34-364E-450C-81AC-D6E4A2E6018A does not exist. Cannot remove artifacts/sids from all</msg>\n </messages>\n</response>\n"
It happens on all nodes, but more frequent when 2 out of 5 are captains:
xxa.med.umich.edu:8089 52
xxb.med.umich.edu:8089 3253
xxc.med.umich.edu:8089 45
xxd.med.umich.edu:8089 107
xxe.med.umich.edu:8089 3799
