Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4405

Avoid calling pollNoWakeup unnecessarily

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.0.1
    • 0.10.2.0
    • None
    • None

    Description

      In KafkaConsumer:poll, the code always calls "pollNoWakeup", which turns out to be expensive. When max.poll.records=1, for example, that call adds about 50% performance overhead. The code should avoid avoid that function unnecessarily when there are no outstanding prefetches.

      Old JIRA description (discarded because turned out not to be the case):
      ---------------------------------------------------------------------------------------------------
      Now kafka consumer has added max.poll.records to limit the count of messages return by poll().

      According to KIP-41, to implement max.poll.records, the prefetch request should only be sent when the total number of retained records is less than max.poll.records.

      But in the code of 0.10.0.1 , the consumer will send a prefetch request if it retained any records and never check if total number of retained records is less than max.poll.records..

      If max.poll.records is set to a count much less than the count of message fetched , the poll() loop will send a lot of requests than expected and will have more and more records fetched and stored in memory before they can be consumed.

      So before sending a prefetch request , the consumer must check if total number of retained records is less than max.poll.records.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            enothereska Eno Thereska
            ysysberserk ysysberserk
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment