Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDBC SOURCE connector adding extra " and \s to json columns #1411

Open
vinay-bh007 opened this issue Apr 18, 2024 · 0 comments
Open

JDBC SOURCE connector adding extra " and \s to json columns #1411

vinay-bh007 opened this issue Apr 18, 2024 · 0 comments

Comments

@vinay-bh007
Copy link

vinay-bh007 commented Apr 18, 2024

I am using kafka JDBC Source connector to fetch the JSON data from table (using a complex SQL query that generates the JSON data using JSON functions of Postgres DB) and using the JsonConverter to publish to kafka topic.

Since the result would have nested JSON, the data when published to kafka topic is having extra " and \ added by converter. Basically the JSON field is treated as String when converted into JSON.
Is there any support to handle nested JSON in connector?

Expected output:

{"key1":"value1",
"NestedKey": [{"nKey1":"nValue1"},{"nKey2":"nValue2"}]
}

Output produced by connector:

{"key1":"value1",
"NestedKey": "[{\"nKey1\":\"nValue1\"},{\"nKey2\":\"nValue2\"}]"
}

Below is the connector properties:

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=false
value.converter.schemas.enable=false

Below is the connector properties:

    "name" : "source-connector",
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "topic.prefix" : "<Name>",
    "connection.url": "jdbc:postgresql://localhost:5432/<DbName>",
    "connection.user" : "<User>",
    "connection.password": "<PW>",
    "poll.interval.ms" : 3600000,
    "query": "<Custom Query having json datatype result>",
    "mode": "bulk",
    "transforms":"createKey, RenameField",
    "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey",
    "transforms.createKey.fields":"tpnb",
    "transforms.RenameField.type":"org.apache.kafka.connect.transforms.ReplaceField$Value",
    "transforms.RenameField.renames":"<Few renames>",
    "db.timezone": "UTC",
    "batch.max.rows": "1000",
    "tasks.max" : "1"
@vinay-bh007 vinay-bh007 changed the title JDBC SOURCE connector adding extra "s to json columns JDBC SOURCE connector adding extra " and \s to json columns Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant