Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7738

Track partition leader epochs in client metadata

Agile BoardAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • None
    • None

    Description

      The Metadata API now exposes the current leader epoch of each partition. We can leverage this information to be smarter in how we fetch metadata. For example, when we receive a NOT_LEADER_FOR_PARTITION, we know to look for an epoch bump before resuming whatever operation we were trying to do. Additionally, this gives us a way to detect stale metadata.

      This requires a little more sophistication in how we track metadata in the client. For example, we may be not be able to assume metadata updates are monotonic. In other words, they may mix stale metadata for one partition and fresh metadata for another. I am not sure we have any strong guarantees on the order in which metadata updates are seen on each broker.

      It may be helpful in this context to control the topics that we fetch metadata for at a finer granularity. Potentially we could even extend the Metadata API to specify a subset of the partitions that the client is interested in.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mumrah David Arthur
            mumrah David Arthur
            Votes:
            0 Vote for this issue
            Watchers:
            3 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment