Hello All, I am testing the upgrade from ES 6.2.0 to 6.6.2.  When I do the upgrade it fails with OSError type 28 no space left of device.  But there is almost 30GB of disk space free.     2021-10-04 19:18:28,028 INFO [615bb5deed7f2dc4595650] _cplogging:216 - [04/Oct/2021:19:18:28] HTTP Request Headers: Remote-Addr: 127.0.0.1 TE: chunked HOST: splunk-sh1.wv.mentorg.com:8000 ACCEPT-ENCODING: gzip, br CACHE-CONTROL: max-age=0 SEC-CH-UA: "Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93" SEC-CH-UA-MOBILE: ?0 SEC-CH-UA-PLATFORM: "Windows" UPGRADE-INSECURE-REQUESTS: 1 ORIGIN: null USER-AGENT: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36 ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 SEC-FETCH-SITE: same-origin SEC-FETCH-MODE: navigate SEC-FETCH-USER: ?1 SEC-FETCH-DEST: document ACCEPT-LANGUAGE: en-US,en;q=0.9 COOKIE: splunkweb_csrf_token_8000=[REDACTED]5649; session_id_8000=[REDACTED]5b74; token_key=[REDACTED]5649; experience_id=[REDACTED]b0c2; splunkd_8000=[REDACTED]tgchx REMOTE-USER: admin X-SPLUNKD: SKdIpkhtf8PlfUDwvOLunA== 11626949294704615649 ijbs1HY^4Ms541EE5sF6eqHg^iyD5t6QKZRByWhdMDXkj546^eB1lT6y59b9LewgHbLcz0Xa5SKotHijcl__zWhYqh8MZISrCqYVxuLkY7jijwyyXijSUQ9VAJRlcQA3o7tgchx 0 Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryO0HdVIPxgJr5HUZN Content-Length: 675766277 2021-10-04 19:18:28,029 INFO [615bb5deed7f2dc4595650] error:333 - POST /en-US/manager/appinstall/_upload 127.0.0.1 8065 2021-10-04 19:18:28,029 INFO [615bb5deed7f2dc4595650] error:334 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. 2021-10-04 19:18:28,029 ERROR [615bb5deed7f2dc4595650] error:335 - Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond self._do_respond(path_info) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 680, in _do_respond self.body.process() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 982, in process super(RequestBody, self).process() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 559, in process proc(self) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 225, in process_multipart_form_data process_multipart(entity) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 217, in process_multipart part.process() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 557, in process self.default_proc() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 717, in default_proc self.file = self.read_into_file() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 732, in read_into_file self.read_lines_to_boundary(fp_out=fp_out) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 702, in read_lines_to_boundary fp_out.write(line) OSError: [Errno 28] No space left on device           As you can see there should be plenty of room for a 670MB upload     splunk@splunk-sh1:~/var/log/splunk> df -kh /opt/splunk Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-splunk 74G 44G 27G 63% /opt splunk@splunk-sh1:~/var/log/splunk>     Web.conf     splunk@splunk-sh1:~/var/log/splunk> more ~/etc/system/local/web.conf [settings] login_content = <h1> <CENTER>Splunk Dev Search Head</CENTER> </h1> max_upload_size = 1024 enableSplunkWebSSL = 1 privKeyPath = /opt/splunk/etc/auth/splunkweb/com.key caCertPath = /opt/splunk/etc/auth/splunkweb/expJun2022.crt splunkdConnectionTimeout = 1400 tools.sessions.timeout = 180 sslVersions = ssl3,tls cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH splunk@splunk-sh1:~/var/log/splunk>     So I am confused why it would say that there is no space left of the device. Thanks ed ... View more

Hello All, I am trying to clean up our indexes and their sizes to ensure that we are keeping the correct amount of data for each index.  I have about 5 to 10 really busy indexes that bring in most of the data.   pan_logs ~200GB/day syslog ~10GB/day checkpoing (coming soon) ~250GB/day  wineventlog ~650GB/day network ~180GB/day So question is if when I create an index configuration for example wineventlog     [wineventlog] homePath = volume:hot/wineventlog/db homePath.maxDataSizeMB = 19500000 coldPath = volume:cold/wineventlog/colddb coldPath.maxDataSizeMB = 58500000 thawedPath = /splunk/cold/wineventlog/thaweddb maxHotBuckets = 10 maxDataSize = auto_high_volume maxTotalDataSizeMB = 78000000 disabled = 0 repFactor=auto     So 30 days of hot/warm would be 1.95TB and 90days of cold data would be 5.85TB and the total size would be 78TB data.  The sizes would then be divided by the total number of indexers we have (20) and each indexer should host about 975GB of hot/warm and 2.925TB of cold data.  And Splunk would start to roll data to frozen (dev null) when the max total (Hot/Warm + Cold) data reached 78TB.  Is that correct? Do I need to specify maxTotalDataSizeMB if I am using homePath and coldPath settings?   Thanks ed ... View more

Hello All, I have configured TA-MS-defender and we are collecting ATP logs just fine.  But the Incident logs keep giving me the following error:   2021-06-25 09:36:35,832 ERROR pid=4306 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events. Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-MS_Defender/bin/ta_ms_defender/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/TA-MS_Defender/bin/microsoft_365_defender_incidents.py", line 72, in collect_events input_module.collect_events(self, ew) File "/opt/splunk/etc/apps/TA-MS_Defender/bin/input_module_microsoft_365_defender_incidents.py", line 69, in collect_events incidents = azutil.get_atp_alerts_odata(helper, access_token, incident_url, user_agent="M365DPartner-Splunk-M365DefenderAddOn/1.3.0") File "/opt/splunk/etc/apps/TA-MS_Defender/bin/azure_util/utils.py", line 57, in get_atp_alerts_odata raise e File "/opt/splunk/etc/apps/TA-MS_Defender/bin/azure_util/utils.py", line 40, in get_atp_alerts_odata r.raise_for_status() File "/opt/splunk/etc/apps/TA-MS_Defender/bin/ta_ms_defender/aob_py3/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://222.178.203.72:19005/whst/63/=Zohzrdbtqhsxzlhbqnrneszbnl//api/incidents?$filter=lastUpdateTime+gt+2000-01-01T00:00:00Z   Any ideas or help?   thanks ed ... View more

Hello All   I added our ES SHC to our monitoring console and the instance(host) name is all the same for all 3 search head cluster nodes.  The instance host (servername) are all unique.  How do I resolve this issue? thanks ed ... View more

Is it possible to use multiple wildcards in the host:: stanza in the props.conf file? [host::svr-*-blah-*] TRANSFORMS-remove = remove_stuff So we are trying to remove stuff from multiple hosts in different geographical locations that have very similar names svr-us-blah-01 svr-us-blah-02 svr-us-blah-03 svr-eur-blah-01 svr-eur-blah-02 svr-eur-blah-03 svr-pac-blah-01 svr-pac-blah-02 svr-pac-blah-03 Each host will collect very similar logs and then forward the logs to Splunk, but we want to dump the noise, so I was hoping that I could just use the [host::svr--blah-] stanza to apply the same props/transforms to each host for dumping the noise. Will that work? thanks ed ... View more