Performance issues with catalog_data_exporter_products indexer

[jochhop]

There's a performance issue with the module magento/module-catalog-data-exporter, it takes more than 1 hour on processing catalog_data_exporter_products indexer, when the reindex happen through cron, the indexer takes so long that it puts in backlog other indexers scheduled after this one, see below image:

The issue seems to be related to out of memory issues when a few crons are scheduled at similar time to catalog_data_exporter_products indexer

Preconditions (*)

1. Magento Cloud 2.4.3-p2 (it happens also in version 2.4.6, but we have widely tested in 2.4.3-p2)
2. 40000 skus, 20 stores
3. Install Adobe Product Recommendations (https://experienceleague.adobe.com/docs/commerce-merchant-services/live-search/guide-overview.html) and/or Adobe Live Search (https://experienceleague.adobe.com/docs/commerce-merchant-services/live-search/guide-overview.html)

Steps to reproduce (*)

1. Have all indexers in mode: Update by schedule
2. Having cron running indexers, from time to time there'll be memory issues and catalog_data_exporter_products will keep status = Working and multiple items in backlog (the issue specially could happen after massive products changes, eg. adding a catalog price rule which affects multiple products for multiple stores)

Expected result (*)

1. Indexers should be processed correctly by cron and in a not escessive amount of time (at least catalog_data_exporter_products should be executed in much less than 1 hour in order not to backlog next scheduled indexers)

Actual result (*)

1. catalog_data_exporter_products will keep status = Working and multiple items in backlog, see below screenshot

[igorwulff]

We've had similar issues that were caused by an badly written query. Could be the same? #24
I'm not sure how versions are exactly handled with commerce releases, but could be worth a check? It is also fixed in the master branch for quite a while.

[realadityayadav]

@jochhop
The product team has recently done a performance improvement on data exporter indexation and syncing. The improvements were released in the latest commerce data exporter version [ 103.1.5 ]. It seems that you are not in the latest commerce data exporter version.
This helped us fix the problem of no results besing send to live search. Please update live-search with --with-dependencies flag to get this fixed.