[Kendra] Jira connector not syncing

[ssmails]

Describe the bug

Created JIRA connector following this documentation https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kendra/client/create_data_source.html

datasource created, but the sync fails. No logs on kendra for the error.
create and sync from kendra UI with same configs as the above failed case, works ok. so seem like boto3 kendra compatibility issue.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

jira sync should work

Current Behavior

datasource created, but the sync fails. No logs on kendra for the error.
create and sync from kendra UI with same configs as the above failed case, works ok. so seem like boto3 kendra compatibility issue.

Reproduction Steps

    def create_new_data_source_jira(self, index_id: str):
        """
        Creates a new Kendra data source based on the configuration provided in the YAML file.

        Returns:
        str: The ID of the created data source, or the ID of the existing data source if one already exists.
        """
        data_source_config = self.config['data_source']

        logger.info(f"role ARN={data_source_config['configuration']['role_arn']}")
        logger.info(f"indexid={data_source_config['indexid']}")

        try:
            response = self.kendra.create_data_source(
                RoleArn=data_source_config['configuration']['role_arn'],
                Name=data_source_config['name'],
                IndexId=data_source_config['indexid'],
                Type=data_source_config['type'],
                Configuration={
                    'JiraConfiguration': {
                        'JiraAccountUrl': 'working jira url which works from kendra ui',
                        'SecretArn': 'working secret art which works from kendra ui',
                        'IssueSubEntityFilter': ['COMMENTS','ATTACHMENTS','WORKLOGS'],
                        'IssueType': ['BUG','STORY','TASK','EPIC'],
                        'AttachmentFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'CommentFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'IssueFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'ProjectFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'WorkLogFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                    }
                }
            )

            data_source_id = response['Id']
            logger.info(f"Data source created with ID: {data_source_id}")

            return data_source_id
        except Exception as e:
            logger.error(f"Error creating data source: {str(e)}")
            raise KendraAdapterException(f"Error creating data source: {str(e)}")

    def start_ingestion_jira(self, index_id, data_source_id):
        try:
            response = self.kendra.start_data_source_sync_job(
                Id=data_source_id,
                IndexId=index_id
            )

            sync_job_id = response['ExecutionId']
            logger.info(f"Data source sync job started with ID: {sync_job_id}")
        except Exception as e:
            logger.error(f"Error starting data source sync job: {str(e)}")
            raise KendraAdapterException(f"Error starting data source sync job: {str(e)}")

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.35.16

Environment details (OS name and version, etc.)

Mac

[tim-finnigan]

Thanks for reaching out. Can you provide more details regarding the sync failure? What error are you getting? The create_data_source command makes a request to the CreateDataSource API, so we'll need more information to investigate if there is some issue with the underlying API behavior.

Can you share a complete code snippet for reproducing the behavior, as well as debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script?

[ssmails]

@tim-finnigan Thanks, unfortunately, there seems to be no error in cloud watch for kendra for this sync failure.

It is simple to reproduce.

1. use boto3 to create a jira connector with the following jira configuration.

                    'JiraConfiguration': {
                        'JiraAccountUrl': 'working jira url which works from kendra ui',
                        'SecretArn': 'working secret art which works from kendra ui',
                        'IssueSubEntityFilter': ['COMMENTS','ATTACHMENTS','WORKLOGS'],
                        'IssueType': ['BUG','STORY','TASK','EPIC'],
                        'AttachmentFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'CommentFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'IssueFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'ProjectFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                        'WorkLogFieldMappings': [
                            {
                                'DataSourceFieldName': 'url',
                                'IndexFieldName': '_source_uri'
                            },
                        ],
                    }

3. sync the datasource created - > the sync fails

[tim-finnigan]

Can you share a complete code snippet for reproducing the behavior, as well as debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script? I'd like to see the specific API response that is being returned here.

[ssmails]

sharing code snippet for the sync failure + logs requested. @tim-finnigan

Note - Datasource was pre-created with 'JiraConfiguration' provided above, using the boto3 sdk. (That operation was successful)

jira-boto3-log-failure.txt

import time
import logging
import boto3
boto3.set_stream_logger('')
from botocore.client import ClientError

def jira_ingest():
    try:
        logging.debug("START.")

        kendra = boto3.client(
            'kendra'
        )
        logging.info("Kendra client initialized successfully.")
        
        response = kendra.start_data_source_sync_job(
            Id="mydatssourceid",
            IndexId="myindexid"
        )
        sync_job_id = response['ExecutionId']
        logging.debug(f"Data source sync job started with ID: {sync_job_id}")
        time.sleep(600)

    except Exception as e:
        logging.error(f"Error: {str(e)}")


if __name__ == '__main__':
    jira_ingest()

[tim-finnigan]

Thanks for following up — from your logs I see that a sync job was started and there is not error or failure present. When you say the sync fails, could you share more details on the failure?

Here in the Kendra Developer Guide is a troubleshooting section on sync jobs failing: https://docs.aws.amazon.com/kendra/latest/dg/troubleshooting-data-sources.html#troubleshooting-data-sources-failed. Have you tried going through the steps documented there

[ssmails]

The sync shows failed in Kendra without any errors.
I have consulted the developer guide already. Want to mention again that using the same role and policy and other configurations, when I try manually - it works form kendra UI.
So, seems like an issue with boto3 library for jira. I have provide the code snippet to reproduce the issue as well, as requested earlier.

[ssmails]

@tim-finnigan appreciate your quick response on this. All requested information has been provided. Thanks.