@edoardo_vicendo we are facing the same issue, but I see the same error even after adding the proxy config under server.conf.. ERROR S2SOverHttpOutputProcessor - HTTP 502 Bad Gateway here's my outputs.conf file.. [httpout] httpEventCollectorToken = ###khldkhfkahl979797#### uri = https://10.x.x.x:443 batchSize = 32768 batchTimeout = 10 it's a network load balancer on AWS, are you using the same kind of load balancer.?? ... View more

which splunk instance you have this configs deployed..?? heavy forwarder or indexer..?? ... View more

@apider Can you give this a try along with other configs in your props.conf TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N TIME_PREFIX=\"time\"\:\s\" MAX_TIMESTAMP_LOOKAHEAD=30 ... View more

@chriswong187 looks like you have to make some tweaks to bypass syslog(in built splunk sourcetype) parsing according to your requirements.. Check this out.... https://www.splunk.com/blog/2008/04/16/overriding-default-syslog-host-extraction.html Check your props/transforms with btool.. $SPLUNK_HOME/bin/splunk cmd btool props list --debug syslog $SPLUNK_HOME/bin/splunk cmd btool props list --debug fml:log ... View more

@vrmandadi best option is to put your HEC's behind a load balancer(F5,ngingx,ha-proxy..etc) ... View more

@sduraisamy : best practice is to NOT touch any file under /etc/system/default, you can either make changes to /etc/system/local or create a custom-app if needed... Above configs do not work on forwarders, you should configure them on indexers... this should work on your indexers... props.conf [appcustom] SHOULD_LINEMERGE = true NO_BINARY_CHECK = true TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N TIME_PREFIX = ^\[\#\| MAX_TIMESTAMP_LOOKAHEAD = 23 -if you think splunk is closing the file while the log is still updating, try something like this in your inputs.conf on your forwarder... time_before_close = <integer> * The amount of time, in seconds, that the file monitor must wait for modifications before closing a file after reaching an End-of-File (EOF) marker. * Tells the input not to close files that have been updated in the past 'time_before_close' seconds. * Default: 3. https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf ... View more

@sduraisamy : firstly, do not make any changes to /etc/system/default, try to use /etc/system/local. do you have the props on your indexers as well..?? ... View more

what's your workflow, do yo have this config on you Splunk-HEC server..?? ... View more

@Kieffer87 : did you try using TIME_FORMAT in your configs... MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %Y-%m-%dT%H:%M:%S%Z ... View more

@ChrisBell04 : how's your props and transforms look like..?? run this to check for any invalid configs$SPLUNK_HOME/bin/splunk btool check ... View more

@luis290311: were you able to fix this issue, we are seeing this on our search-heads, we are 7.2.1 ... View more

@suarezry : did you look at this f5 article related to that error... https://support.f5.com/csp/article/K17398511 ... View more

@raindrop18: excess copies result from peers(indexers) leaving and joining the cluster. you can remove excess buckets from cluster master via cli or web ui. follow the splunk doc for further explanation.. https://docs.splunk.com/Documentation/Splunk/7.2.1/Indexer/Removeextrabucketcopies also take a look at data-rebalancing, cluster master removes excess buckets during data-rebalance.. https://docs.splunk.com/Documentation/Splunk/7.2.1/Indexer/Rebalancethecluster#How_data_rebalancing_works ... View more

@kamal_jagga : If you have a DMC configured in your environment, there are prebuilt dashboards in there... DMC-->search--->activity--->search activity:instance--->select your SearchHead instance(Top 20 Memory-Consuming Searches if you scroll down). you can use that underlying search with some tweaks to build a dashboard. https://docs.splunk.com/Documentation/Splunk/7.2.1/DMC/SearchactivityDeploymentwide#Interpret_results_in_these_dashboards ... View more

There are few configs that are not recommended to distribute through the bundle... https://docs.splunk.com/Documentation/Splunk/7.2.1/Indexer/Updatepeerconfigurations#Settings_that_you_should_not_distribute_through_the_configuration_bundle How the file precedence works.. http://docs.splunk.com/Documentation/Splunk/7.2.1/Admin/Wheretofindtheconfigurationfiles#Precedence_for_cluster_peer_nodes Coming to conflicts, it depends on your orchestration when you make any changes to configs. ... View more

check your inputs.conf on your syslog(do you have any host_segement or host_regex in there).. index=network | dedup host | table host (might give you hosts forwarding to that index) ... View more

@arunprasath93: looks like you were able to post the message successfully, did you do a all-time search just in case if the timestamps are off, if you have associated any index to your HEC token, try running a run with your index and Sourcetype. ... View more

you will end up with all configs in one location($SPLUNK_HOME/etc/master_apps/_cluster/local), but with custom-apps... for instance I create 2 different custom-apps like network_TA(props and transforms..etc)for my network gear,apache_TA(props and transforms..etc) for apache logs.. In that way it's easy to manage the configs based on functionality, at the end it's your preference 🙂 ... View more

@rung8: I would do this, create a custom-app on your cluster-master and apply the bundle, in this way you have more control over the configurations you deploy and it's easy to manage... 1. ClusterMaster: $SPLUNK_HOME/etc/master-apps/customapp(with all configs in here) NOTE:you can have multiple-custom apps based on the functionality(easy to differentiate and troubleshoot) 2. ClusterMaster:apply cluster bundle 3. Indexers(Peer-nodes): $SPLUNK_HOME/etc/slave-apps/customapp(downloaded here by default) Follow this splunkdoc for more details... https://docs.splunk.com/Documentation/Splunk/7.2.1/Indexer/Updatepeerconfigurations#Structure_of_the_configuration_bundle ... View more

@qtorque95 : looks like you have Splunk-enterprise installed on your local... 1.try running this command to check the inputs status of the monitor path $SPLUNK_HOME/bin/splunk list input status 2. if you see your monitor path from the list above, you can reset the file checkpoints(splunk might be thinking the above file as a duplicate) https://docs.splunk.com/Documentation/Splunk/7.2.1/Troubleshooting/CommandlinetoolsforusewithSupport#btprobe read this splunk doc on How Splunk calculates CRC.. https://docs.splunk.com/Documentation/Splunk/7.2.1/Data/Howlogfilerotationishandled 3. Stop Splunk, delete fishbucket($SPLUNK_HOME/var/lib/splunk/fishbucket), and start splunk(this will reindex all files, NOT a best solution on prod boxes) ... View more

@kaumiladani : by default macros are not distributed to indexers in a clustered environment, did you try adding this stanza in your apps/TA's: default/distsearch.conf [replicationSettings:refineConf] replicate.macros = true OR you can get rid of macros from event types, and create a local copy. https://answers.splunk.com/answers/661233/error-searchparser-the-search-specifies-a-macro-cs.html ... View more

@yzaari: let's assume that your index=network, there are many ways to grab the info, I will list few here... | metadata type=hosts index=network | tstats values(host) as hosts, values(sourcetype) as sourcetypes where index=network | tstats values(sourcetype) values(host) where index=network group by index https://docs.splunk.com/Documentation/Splunk/7.2.1/SearchReference/Metadata ... View more

@ikaneng: how's your raw data look like, we need more details to come up with a search... If it's a ipv4 you can have this in your base search, you might have to use cidrmatch for ipv6... e.g: index=index_name sourcetype=stype subnet_ip=10.0.0.1/24 | stats count, max(connsbyHost) as max_bandwidth, min(connsbyHost) as min_bandwidth, avg(connsbyHost) as avg_bandwidth BY Interface go though this splunk docs for reference.. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Aggregatefunctions#max.28X.29 http://docs.splunk.com/Documentation/Splunk/7.2.1/SearchReference/ConditionalFunctions#cidrmatch.28.22X.22.2CY.29 ... View more

@Log_wrangler : did you setup your alert to have PDF as an attachment..?? I would check index=_internal source=*pdfgen.log and index=_internal source=*python.log to get more details about your scheduled report. ... View more