Hello!   We have some logs coming across which are in JSON and thus 'just work'. The problem is, inside the log field are the events we need to extract.  There are about 200 app's that will be logging this way and each app will have different fields and values so doing a standard field extract won't work or would be 1000's of potential kvps. The format however is always the same.     { "log": "[2022-08-25 18:54:40.031] INFO JsonLogger [[MuleRuntime].uber.143312: [prd-ops-mulesoft].encryptFlow.BLOCKING @4ac358d5] [event: 25670349-e6b5-4996-9cb6-c4c9657cd9ba]: {\n \"correlationId\" : \"25670349-e6b5-4996-9cb6-c4c9657cd9ba\",\n \"message\" : \"MESSAGE_HERE\",\n \"tracePoint\" : \"END\",\n \"priority\" : \"INFO\",\n \"elapsed\" : 0,\n \"locationInfo\" : {\n \"lineInFile\" : \"95\",\n \"component\" : \"json-logger:logger\",\n \"fileName\" : \"buildLoggingAndResponse.xml\",\n \"rootContainer\" : \"responseStatus_success\"\n },\n \"timestamp\" : \"2022-08-25T18:54:40.030Z\",\n \"content\" : {\n \"ResponseStatus\" : {\n \"type\" : \"SUCCESS\",\n \"title\" : \"Encryption successful.\",\n \"status\" : \"200\",\n \"detail\" : { },\n \"correlationId\" : \"25670349-e6b5-4996-9cb6-c4c9657cd9ba\",\n \"apiMethodName\" : null,\n \"apiURL\" : \"https://app.com/prd-ops-mulesoft/encrypt/v1.0\",\n \"apiVersion\" : \"v1\",\n \"x-ConsumerRequestSentTimeStamp\" : \"\",\n \"apiRequestReceivedTimeStamp\" : \"2022-08-25T18:54:39.856Z\",\n \"apiResponseSentTimeStamp\" : \"2022-08-25T18:54:40.031Z\",\n \"userId\" : \"GID01350\",\n \"orchestrations\" : [ ]\n }\n },\n \"applicationName\" : \"ops-mulesoft\",\n \"applicationVersion\" : \"v1\",\n \"environment\" : \"PRD\",\n \"threadName\" : \"[MuleRuntime].uber.143312: [prd-ops-mulesoft].encryptFlow.BLOCKING @4ac358d5\"\n}\n", "stream": "stdout", "time": "2022-08-25T18:54:40.086450071Z", "kubernetes": { "pod_name": "prd-ops-mulesoft-94c49bdff-pcb5n", "namespace_name": "4cfa0f08-92b0-467b-9ca4-9e49083fd922", "pod_id": "e9046b5e-0d70-11ed-9db5-0050569b19f6", "labels": { "am-org-id": "01a4664d-9e16-454b-a14c-59548ef896b5", "app": "prd-ops-mulesoft", "environment": "4cfa0f08-92b0-467b-9ca4-9e49083fd922", "master-org-id": "01a4664d-9e16-454b-a14c-59548ef896b5", "organization": "01a4664d-9e16-454b-a14c-59548ef896b5", "pod-template-hash": "94c49bdff", "rtf.mulesoft.com/generation": "aab6b8074cf73151b1515de0e468478e", "rtf.mulesoft.com/id": "18d3e5d6-ce59-4837-9f3b-8aad3ccffcef", "type": "MuleApplication" }, "host": "1.1.1.1", "container_name": "app", "docker_id": "cf07f321aec551b200fb3f31f6f1c67b2678ff6f6a335d4ca41ec2565770513c", "container_hash": "rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.3.0@sha256:6cfeb965e0ff7671778bc53a54a05d8180d4522f0b1ef7bb25e674686b8c3b75", "container_image": "rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.3.0:20211222-2" } }       The JSON works fine, but the events we ALSO want extracted are in here      {\n \"correlationId\" : \"25670349-e6b5-4996-9cb6-c4c9657cd9ba\",\n \"message\" : \"MESSAGE_HERE\",\n \"tracePoint\" : \"END\",\n \"priority\" : \"INFO\",\n \"elapsed\" : 0,\n \"locationInfo\" : {\n \"lineInFile\" : \"95\",\n \"component\" : \"json-logger:logger\",\n \"fileName\" : \"buildLoggingAndResponse.xml\",\n \"rootContainer\" : \"responseStatus_success\"\n },\n \"timestamp\" : \"2022-08-25T18:54:40.030Z\",\n \"content\" : {\n \"ResponseStatus\" : {\n \"type\" : \"SUCCESS\",\n \"title\" : \"Encryption successful.\",\n \"status\" : \"200\",\n \"detail\" : { },\n \"correlationId\" : \"25670349-e6b5-4996-9cb6-c4c9657cd9ba\",\n \"apiMethodName\" : null,\n \"apiURL\" : \"https://app.com/prd-ops-mulesoft/encrypt/v1.0\",\n \"apiVersion\" : \"v1\",\n \"x-ConsumerRequestSentTimeStamp\" : \"\",\n \"apiRequestReceivedTimeStamp\" : \"2022-08-25T18:54:39.856Z\",\n \"apiResponseSentTimeStamp\" : \"2022-08-25T18:54:40.031Z\",\n \"userId\" : \"GID01350\",\n \"orchestrations\" : [ ]\n }\n },\n \"applicationName\" : \"ops-mulesoft\",\n \"applicationVersion\" : \"v1\",\n \"environment\" : \"PRD\",\n \"threadName\" : \"[MuleRuntime].uber.143312: [prd-ops-mulesoft].encryptFlow.BLOCKING @4ac358d5\"\n}     SPATH would work, but the JSON is fronted by this     "[2022-08-25 18:54:40.031] INFO JsonLogger [[MuleRuntime].uber.143312: [prd-ops-mulesoft].encryptFlow.BLOCKING @4ac358d5] [event: 25670349-e6b5-4996-9cb6-c4c9657cd9ba]:     So it doesn't treat it as JSON. This is one example, others events don't have the correlationID, etc.  We need a method that will take the raw data, parse it as JSON AND then dynamically extract the events in the log field in their KV pairs (e.g. \"correlationId\" : \"25670349-e6b5-4996-9cb6-c4c9657cd9ba\" == $1::$2) Can this be done in transforms using regex? Is this even possible, or do we ultimately need to create extractions based on every possible field?   Appreciate the guidance here! Thanks!! ... View more

Hello Splunkers, I have a bit of an issue onboarding some AWS Canaries from S3.  We have deployed the SQS/SNS and S3 and the files are coming in fine.  However each canary writes 4 files: TXT - detailed report -  I want this JSON - summary report - I want this PNG - image - I don't want this HTML - I don't want this Because they are all coming on a single queue they are getting the same sourcetype which is not working as they are very different structures.  As such I've built the following props: [aws:canaries:summary] DATETIME_CONFIG = INDEXED_EXTRACTIONS = json KV_MODE = none LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Structured description = AWS Canary JSON Summary File disabled = false pulldown_type = true [aws:canaries:detailed] BREAK_ONLY_BEFORE_DATE = DATETIME_CONFIG = LINE_BREAKER = (Start Canary) MAX_TIMESTAMP_LOOKAHEAD = 50 NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_PREFIX = timestamp\:\s TRUNCATE = 0 category = Application description = AWS Canary Detailed TXT file disabled = false pulldown_type = true This is on the HF and the Indexer cluster. In order to separate and drop the files the following is also added to the props/transforms: Props: [aws:canaries] TRANSFORMS-aws_canaries = set_aws_canary_json, set_aws_canary_txt, drop_aws_canary_png, drop_aws_canary_html Transforms: ##### AWS Canaries - change sourcetype for SQS based S3 files and drop PNG files ##### [set_aws_canary_json] SOURCE_KEY = MetaData:Source REGEX = ^source::.*json FORMAT = sourcetype::aws:canaries:summary DEST_KEY = MetaData:Sourcetype [set_aws_canary_txt] SOURCE_KEY = MetaData:Source REGEX = ^source::.*txt FORMAT = sourcetype::aws:canaries:detailed DEST_KEY = MetaData:Sourcetype [drop_aws_canary_png] SOURCE_KEY = MetaData:Source REGEX = ^source::.*png FORMAT = nullQueue DEST_KEY = queue [drop_aws_canary_html] SOURCE_KEY = MetaData:Source REGEX = ^source::.*html FORMAT = nullQueue DEST_KEY = queue Again, both applied to HF and Indexers. On the inputs I have the following: [aws_sqs_based_s3://canaries] aws_account = SplunkForwarderRole aws_iam_role = canaries index = canaries interval = 300 s3_file_decoder = CustomLogs sourcetype = aws:canaries sqs_batch_size = 10 sqs_queue_region = us-east-1 sqs_queue_url = http://222.178.203.72:19005/whst/63/=ptdtdzZlZynmZvrzbnl//1111111111/canaries disabled = 1   The idea being that the input receives the data and sourcetypes it aws:canaries then parsing/transorms alters the sourcetype. On the SH I can see the files are being source typed correctly (and dropped) however event breaking is not working... the JSON is everyline and it looks like the TXT file breaks at the date. Anyone configured something similar? I suspect the event breaking is happening as part of the aws:canaries sourcetype?  Just not sure Any help appreciated!! ... View more

Hello Splunkers, We are receiving config notifications, CloudTrail and others from AWS through Kinesis - the general pattern is: Config -> Event Rules -> Event Hub -> Kinesis -> HEC (indexer cluster) This works, what seems flawlessly. We separate the data into indexes based on the account number due to security policies so I've set up props/transforms to do this. This seems to work most of the time and 77% of the traffic ends up in the correct index but there is 33% (on avg) that ends up in the default index of the HEC (aws_config) Is it possible that the transforms aren't triggering all the time? The events are identical format; sourcetype and source are identical. Here is the transforms: [aws-account1] REGEX = 010016492034 DEST_KEY = _MetaData:Index FORMAT = aws-account1 Props: [aws:config:notification] TRANSFORMS-aws_config_notification=aws-account1 Am I missing something here? Is there anything I should look for in internal? I remember years ago in training an example where props hierarchy would mess with data when there was multiple props/transforms, and the intermittent nature *might* make sense but I have no idea where to troubleshoot this. The HEC is on a cluster of indexers so the config is all via the CM, thus no differences. Any suggestions would be greatly appreciated! Cheers! ... View more