Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Currently, relRecord is populated with cached schema only at the top-level fields while the nested fields are populated with the schema of the incoming record. This ticket is to fix that inconsistency and always consider cached schema (schema that is fetched when the job starts).
As a by-product of considering only cached schema, Sql job will keep running even though the input schema has evolved, let's say a new field is added. We will keep dropping the new field. As a separate effort, we need to consider adding metrics to flag whenever we detect that the input schema has evolved.
Attachments
Issue Links
- links to