Optimize Your Search with Partitions

What is a Partition?

manage variable retention, and specify certain data to forward to S3.

Partitions route your data to an index becoming a separate subset of data in your account. Creating smaller and separate subsets of data is central to search optimization. When you run a search against an index, results are returned more quickly and efficiently because the search runs against a smaller data set.

This example shows a customer that created three additional Partitions to separate data by environment.

Consider the following queries:

Query	Partition Status	Path
Query 1	Â	_sourceCategory=prod/security/snort
Query 2	Partitions in place	_index=prod AND _sourceCategory=prod/security/snort
Query 3	Partitions in place	_sourceCategory=prod/security/snort
Query 4	Partitions in place	_sourceCategory=stage/aws/cloudtrail OR _sourceCategory=prod/security/snort

• Query 1. There are no custom Partitions created and you only have the Default Index, 100% of your data across all partitions is scanned in order to find all production log messages for the Snort security app.
• Query 2. Partitions do exist, _index=prod limits the scope of the query and only about 40% of the data is scanned to get the same results as Query 1. But it is redundant.
• Query 3. You can take advantage of Partitions without having to rewrite your existing queries. Sumo Logic's behind-the-scenes Query Rewriting, performed for queries run against data, is smart enough to understand that the scope of what you are looking for is included within _index=prod; therefore at runtime, it will rewrite the query as Query 2.
• Query 4. We want to search for data that is in a custom Partition, as well as data that exists in the Default Index. However, query rewriting does not have the ability to OR indexes together. Instead, another behind-the-scenes feature, Inverse View Rewriting kicks in, we know that the data is NOT contained in the DEV and QA index, so those will be skipped. This query will only scan the Prod index and the Default Index.

You can create partitions within both data tier and flex plans.

Data Tier plans

Enterprise Suite Credits account type. By default, all log data ingested into Sumo Logic is assigned to the sumologic_default partition, which operates on the Continuous tier. If you have an Enterprise Suite Credits account type, you have the option to assign some log data to either the Frequent or Infrequent tier by creating a separate partition with a specific routing expression and assigned data tier.

Different data tiers come with varying credit burn rates. Costs will be incurred for ingesting log data into Sumo Logic based on the tier: Continuous, Frequent, and Infrequent. Logs held in the Infrequent tier will also incur costs when the data is subsequently queried for analysis. The number of credits consumed will be determined by the volume of log data scanned. To learn more, refer to the Data Tier Partitions.

Flex plans

Flex Partitions.

*Customers licensed for Enterprise Suite Flex can also license Cloud SIEM. With this, the subset of ingested log data forwarded to the Cloud SIEM solution is not categorized as Flex data. Instead of price being based on the volume of data scanned by queries, it is based on the volume of log data forwarded to Cloud SIEM.

What is Query Rewriting?

• _sourceCategory naming convention
• The Partition was scoped using _sourceCategory
• The searches are using _sourceCategory, so they can easily be mapped to Partitions
• The scope of this search (_sourceCategory=prod/security/snort) falls within the scope of the Partition (_sourceCategory=prod/*)

Therefore, defining a broad scope for your Partitions (for example, _sourceCategory=prod/*), and searching with _sourceCategory allows you to take advantage of query rewriting, and it allows you to potentially not have to manually rewrite your existing queries.

note

We have used a simple example of non-overlapping partitions all defined on _sourceCategory. Your data organization needs may be more complex, and in those cases we try to do a best effort query re-writing.

Create a Partition

create a Partition named Dev:

1. Classic UI. In the main Sumo Logic menu, select Manage Data > Logs > Partitions.
   New UI. In the top menu select Configuration, and then under Logs select Partitions. You can also click the Go To... menu at the top of the screen and select Partitions.
2. Click Add Partition.
3. In the Name field, enter Dev.
4. In the Routing Expression field enter _sourceCategory=Dev.
5. Select Apply the retention period of sumologic_default.
6. Click Save.

How can my team use Partitions?

Manage Partitions.