[BUGFIX] Bugfixed in KAFKA-8713, but it doesn't work properly.

[krespo]

Through the ticket "KAFKA-8713", a bug that always outputs "default value" when JsonConverter is a nullable schema was fixed.

After applying the bug fix, I set the following settings in kafka connect.

"value.converter.replace.null.with.default": "false",
However, I found that the same problem still occurred, so I modified the code to fix this bug.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[gharris1727]

@krespo It looks like the replace.null.with.default feature of the JsonConverter is due to be released in 3.5.0, which is not available yet. Are you testing this with 3.4.0 or earlier? In that case, setting the value won't change the behavior of the converter.

You can test this by building and using the json converter jar from the 3.5 branch, or pick up a 3.5.0 release candidate from the mailing list: https://lists.apache.org/thread/0vfdhwvpjhfq4c7p818dtm07hr6j0qqs

[krespo]

@gharris1727
First of all thank you for your feedback.
As you said, I know that feature is only available in kafka version 3.5, which hasn't been released yet.

The open source versions I'm using are as follows:

debezium: 2.1
kafka connect: 2.7.1
mysql: 8

When the column in mysql is "nullable" and "default value" is set, we founded that the message that the kafka connector sends through the JsonConverter is always the "default value", even though the column is updated to "null".

So I checked the contents of "KAFKA-8713", and after building the source code of Kafka 3.5 version and creating "connect-json-3.5.0.jar", the "${kafka_connect_home} /libs" in the connect-json library was replaced with a new version.

Afterwards, I set the "replace.null.with.default" setting to "false", but found that the same problem still occurs.
And through debugging, it was found that to solve the "KAFKA-8713" issue, not only the connect-json module but also the connect-api module had to be modified.
So I made a pull request for my modified code.
Currently, connect-json.jar and connect-api.jar have been replaced with version 3.5, and they are working properly.

Please check the code I modified once again. thank you

[gharris1727]

Hey @krespo

Thank you for clarifying your setup. I was able to reproduce this issue in a unit test. I'll bring it up to the release managers to see if we can get a fix into 3.5. Thanks so much for validating this feature pre-release.

Unfortunately this patch can't move forward as-is, because people are certainly relying on the Struct::get returning defaults. However, we can alter the JsonConverter to use Struct::getWithoutDefault, and use the replace.null.with.default configuration to gate whether to insert the default to maintain backwards compatibility.

If you can update this patch to call getWithoutDefault, and add some unit tests for the struct field case, I think that will be merge-able.

cc @mimaison @C0urante

[mimaison]

Good catch!
Yes it would be good to have this in 3.5. @krespo can you update your PR with the suggestions from @gharris1727 ? Thanks

[mimaison]

@gharris1727 If @krespo is not able to update the PR, can you open one?

[mimaison]

Superseded by #13781