Catalog Synchronization: How It Works and How to Make It Fast

Catalog synchronization looks simple from Backoffice: a button that copies Staged to Online. Under that button sits one of the platform's oldest and most performance-sensitive subsystems, and nearly every mature SAP Commerce project eventually has the same conversation: "why does sync take four hours, and why did it fail at 3 a.m. with 'There were errors during the synchronization!' and no further explanation?" This guide is that conversation, answered properly.

Who Triggers Sync, and Why It Matters#

A synchronization is either triggered manually (a business user in Backoffice) or by a scheduled cronjob. For full catalog syncs, scheduled wins on every axis:

Manual sync still has a place: single-product or category-scoped publishes where a business user needs content live now. The operational rule is that full-version syncs belong to the scheduler, and business users get scoped sync plus training on what the sync button actually does.

On CCv2, pin synchronization jobs to the background processing aspect through node groups. A full sync running on a storefront node is a self-inflicted incident.

Selective Sync: The readyForSync Pattern#

Syncing only approved items is a near-universal requirement, and the instinct to reuse the OOTB approvalStatus attribute is wrong: approval and publication-readiness are different business states, and coupling them means you cannot sync an unapproved-but-ready item or hold back an approved one. Introduce a dedicated boolean:

<attribute qualifier="readyForSync" type="java.lang.Boolean">
  <defaultvalue>Boolean.FALSE</defaultvalue>
  <persistence type="property"/>
</attribute>

For scheduled syncs, enforce it with a search restriction bound to a dedicated sync user group:

INSERT_UPDATE SearchRestriction; code[unique=true]; name[lang=en]; query; principal(UID); restrictedType(code); active; generate
; Backend_Sync_Product ; Sync ; {item.readyForSync}=1 ; syncgroup ; Product ; true ; true

Restrictions are disabled in the default sync session, so a custom sync job has to switch them back on:

public class AcmeCatalogVersionSyncJob extends CatalogVersionSyncJob
{
    @Override
    protected SessionContext createSyncSessionContext(final SyncItemCronJob cronJob)
    {
        final SessionContext ctx = super.createSyncSessionContext(cronJob);
        // enable user and group search restrictions for this sync session
        ctx.setAttribute(FlexibleSearch.DISABLE_RESTRICTIONS, Boolean.FALSE);
        ctx.setAttribute(FlexibleSearch.DISABLE_RESTRICTION_GROUP_INHERITANCE, Boolean.FALSE);
        return ctx;
    }
}

Two caveats that bite teams:

• Restrictions do not apply to manual sync. A business user syncing from Backoffice syncs everything in scope, readyForSync or not. If the flag must be authoritative, scheduled sync is the only publication path and manual full sync gets locked down by permission.
• In multi-team catalogs (different teams owning different brands or categories), combine the flag with category-scoped sync jobs so team A's publication run does not carry team B's half-finished items. The flag gives per-item control; category scoping gives per-team isolation; you want both.

"Why Is My Product Still Out of Sync?" Closely Related Types#

Business users think of a product as one thing. The type system knows it as a product plus prices, media, variants, and stock, each a separate type. When a related item changes, whether the product shows as out-of-sync in the cockpit depends on the relatedReferencesTypesMap of the synchronization service. Media works out of the box; your custom related types will not, until you extend the map:

<alias alias="synchronizationService" name="acmeSynchronizationService"/>
<bean id="acmeSynchronizationService"
      class="de.hybris.platform.cockpit.services.sync.impl.SynchronizationServiceImpl"
      parent="defaultSynchronizationService">
  <property name="relatedReferencesTypesMap">
    <map merge="true">
      <entry key="Product">
        <list>
          <value>Product.productImages</value>
          <value>Product.complianceInfos</value>
        </list>
      </entry>
      <entry key="MediaContainer">
        <list><value>MediaContainer.medias</value></list>
      </entry>
    </map>
  </property>
</bean>

Nested structures (variant under variant under product) do not reliably cascade: the algorithm stops at the first level that reports "in sync". If your model nests deeply, do not promise business users that the out-of-sync indicator is exhaustive.

Part-Of, Untranslatable, and Root Types#

The heart of sync configuration is deciding, per catalog-aware reference attribute, how the target side gets resolved. Simple values copy; catalog-unaware references copy by PK; catalog-aware references need translation to the corresponding target-version item. Two mechanisms exist.

Per-attribute configuration on the SyncAttributeDescriptorConfig, where the interaction of "copy by value" (part-of) and "untranslatable" produces four distinct behaviours worth memorizing:

Root type declaration is the alternative for stand-alone types: full sync iterates root types in declared order and guarantees every source item exists in the target version. Declare a root type when the type has independent existence (it is not merely owned by another item); use part-of for true compositions.

Root type ordering is a performance lever, not a formality: sync resolves references in passes, and a reference to an item whose type has not been processed yet forces another pass. Order custom root types so that referenced types come before referencing types, and multi-pass resolution mostly disappears.

Merging Multiple Sources into One Target#

Fan-in (several staged versions syncing into one online version) works untouched only under specific conditions: each job owns disjoint items, or disjoint attributes, or disjoint languages. Then "last write wins" is harmless because writes never collide.

The collision case is relational attributes, classically product-to-category. Sync replaces the target's product.supercategories with the source's values wholesale, obliterating references added by other syncs, ImpEx, or API calls. When the business rule is "append, do not replace", the clean solution is a customized CatalogVersionSyncJob implementing the merge in CatalogVersionSyncCopyContext.translate(..).

The tempting shortcut, excluding the attribute from sync on one side of the relation and letting the other side carry it, works but inverts editing semantics (changes must then be made on the synchronized side only), and that non-obviousness will confuse every future maintainer. Customize the job; the shortcut's confusion costs more than the code.

Stop Synchronizing Prices and Media#

The single biggest sync accelerator on most projects is realizing that two of the heaviest item families do not belong in sync at all.

Prices. When prices come from ERP or a pricing service, staging them in commerce is pure cost: database bloat, slower syncs, and a nasty coupling where any PriceRow change marks its product as modified, forcing business users to publish product content just to get a price live. Decouple PriceRows from product versions (reference by product code rather than by catalog-aware product reference) and write them straight to where the storefront reads them, after validation. The platform documents this as decoupling PDT rows from products.

Media. Where media conversion and URLs are managed externally, put media in a separate single-version media catalog. Product catalogs then sync without cloning media items, which in a large catalog with several media per product removes an enormous volume from every run. Even with internal media management, an immutable-media policy (new image means new media code, never in-place update) lets staged and online products reference different media directly, keeping an approval flow while bypassing media sync entirely.

Troubleshooting a Failed Sync#

The infamous log line:

[CatalogVersionSyncJob] Finished synchronization in 0d 00h:00m:00s:442ms. There were errors during the synchronization!

usually means untranslatable references: a catalog-aware reference whose target item does not exist, on an attribute that is neither part-of nor pointing at a root type. The systematic path from that line to a fix:

1. Read the sync dump CSV. The job writes a CSV of pending rows: source PK, target PK, sync timestamp PK, and the pending attribute qualifier. That last column names the attribute that failed to translate. For collections, one untranslatable member marks the whole attribute pending.
2. Raise logging when the dump is not enough. Set cronjob.logtofile.threshold=DEBUG and set these loggers to DEBUG (they may only appear in HAC after a sync has loaded the classes on that node):

log4j.logger.de.hybris.platform.catalog.jalo.synchronization.CatalogVersionSyncWorker=DEBUG
log4j.logger.de.hybris.platform.catalog.jalo.synchronization.CatalogVersionSyncMaster=DEBUG
log4j.logger.de.hybris.platform.catalog.jalo.synchronization.AbstractItemCopyContext=DEBUG
log4j.logger.de.hybris.platform.catalog.jalo.synchronization.CatalogVersionSyncCopyContext=DEBUG

1. Read the background-processing node logs. Now they name the real cause, for example:

[...MandatoryAttributesValidator]: missing values for [code] in model ApparelProductModel (8832234909114) to create a new ApparelProduct

1. Find all affected items, not just the reported one:

SELECT {pk} FROM {ApparelProduct} WHERE {code} IS NULL

1. Fix with a script, keep the script. Whether the remedy is a Groovy repair or a removal ImpEx:

REMOVE ApparelProduct; pk[unique=true]
; 8951755421348
; 8951755624411

the script goes into the deployment runbook, because the same corrupt data pattern is waiting in staging and production.

1. Re-run and iterate. Fresh errors surface as earlier ones clear; the loop converges quickly once the dominant cause is fixed.

Performance Tuning, in Order of Impact#

There is no universal items-per-second benchmark; throughput depends on type complexity, relation density, and language count. The working heuristic: 1 to 5 items per second per thread means something is wrong. Well-tuned product syncs run an order of magnitude faster. The levers, roughly in the order to pull them:

1. Threads#

catalog.sync.workers scales nearly linearly until the application server or database saturates. Start at twice the core count of the background processing node (recent versions default sensibly) and watch database load as you raise it. More threads against a saturated database makes things worse, not better.

2. Transactions#

catalog.sync.enable.transactions=true batches each item's attribute writes into one statement flush instead of one UPDATE per attribute set. Equivalent service-layer behaviour comes from model.service.transactional.saves=true, but note that property is global, not sync-specific. Transactions introduce deadlock potential under high parallelism; this lever requires a real load test before production, not a property flip on Friday.

3. Ordered relations#

Ordered relations are sync poison: sequence maintenance turns set operations into row-by-row work. Audit every custom relation and set ordered="false" unless a human being actually curates ordering:

<relation code="Supplier2SupplierProductRel" generate="true" localized="false" autocreate="true">
  <sourceElement qualifier="supplier" type="Supplier" cardinality="one"/>
  <targetElement qualifier="supplierProducts" type="SupplierProduct" cardinality="many" ordered="false"/>
</relation>

The category-product relation is defined in a core extension, but the platform exposes a property for it. If products reach the storefront through Solr anyway (they do), category ordering is usually cosmetic fiction, so:

relation.CategoryProductRelation.source.ordered=false

4. Relation modification marking#

Setting one side of a relation can timestamp-touch every item on the other side: category.setProducts(...) fires an UPDATE per product just to mark it modified. Disabling the marking removes that write storm:

relation.CategoryProductRelation.markmodified=false

Use with eyes open: sync itself uses modification marks to decide what to copy. Disable marking only on relations you have explicitly excluded from sync-relevance.

5. PriceRow product marking#

Every PriceRow attribute setter marks the owning product modified; three attribute changes mean three product UPDATEs. During full syncs this is pointless (PriceRow is its own root type). Suppress it in the sync session:

@Override
protected SessionContext createSyncSessionContext(final SyncItemCronJob cronJob) {
    final SessionContext ctx = super.createSyncSessionContext(cronJob);
    ctx.setAttribute(Europe1Constants.PDTROW_MARK_PRODUCT_MODIFIED, Boolean.FALSE);
    return ctx;
}

If the business genuinely needs price changes to mark products, do it once per product via an interceptor, not once per attribute.

6. Remove attributes from the sync configuration#

Every attribute in the job is work per item, and relation attributes are the expensive ones. Candidates almost every project can drop from product sync:

Item.comments, Item.assignedCockpitItemTemplates, Category.linkComponents, Category.productCarouselComponents, Category.restrictions, Category.productListComponents, Product.linkComponents, Product.restrictions, Product.productCarouselComponents, Product.productListComponents, Product.productDetailComponents, Product.promotions.

Collection-typed attributes deserve extra suspicion: they serialize into a column, change detection does not work properly for them, and they get rewritten even when unchanged. If Product.buyerIDS, Product.specialTreatmentClasses, or Product.articleStatus are unused on your project, remove them from sync. The shipped configuration lives in the productCatalogEditSyncDescriptors bean in commerceservices-spring.xml; note that changed bean configuration only applies when the sync job is created (initialization), so existing jobs need updating through Backoffice or a migration script.

For very high category-to-product cardinality, the default behaviour loads all product PKs of a category into memory and blocks threads on it. If the relation can be maintained from the product side only, remove Category.products from the sync configuration.

7. Sync languages#

Per-country catalogs rarely need every platform language. Set the catalog version's languages so localized attribute copying covers only what the storefront serves. On a type with many localized attributes across 20 configured languages, this is a large multiplier.

8. Initial attributes#

Item creation during sync costs at least two statements (INSERT with initial attributes, UPDATE with the rest), four with localized attributes. Attributes can be promoted into the creation map by overriding isRequiredForCreation(..) in a custom copy context:

@Override
protected CatalogVersionSyncCopyContext createCopyContext(final CatalogVersionSyncCronJob cj,
        final CatalogVersionSyncWorker worker) {
    return new AcmeCatalogVersionSyncCopyContext(this, cj, worker);
}

This is the deepest cut in the list: the attribute must also be initial in the Jalo layer, part-of attributes must stay out of it, and attributes with overridden Jalo setters as well. Reach for it only after levers 1 through 7 are exhausted.

The Database Side#

Sync is ultimately an insert/update firehose, and past a point the bottleneck is below the JVM. Watch two things: database load (CPU, memory, I/O) and the network round-trip between background processing nodes and the database. With JDBC logging enabled, healthy per-statement times are 0 to 2 ms; sustained 10 ms and up points at the database or the link, not at your sync configuration.

• Missing indexes are rare in sync paths but real for custom types with composite unique catalog keys; verify the key attributes are covered by an index. Also relevant: checkCatalogVersionValidity() runs duplicate checks per root type before sync starts, which on a big catalog is expensive; with proper unique indexes in place, overriding the job to skip duplicate checking for specific root types is a legitimate optimization.
• Surplus indexes slow every write; have unused ones identified and dropped.
• Statistics and fragmentation produce the most dramatic numbers in the whole guide. A real case: after importing roughly 500,000 products, one query consumed 80 percent of database time at about 280 ms per execution; rebuilding table statistics took it below 1 ms and moved product sync from 30 items per second to 290. On CCv2 you do not administer Azure SQL yourself; when the symptoms fit (query performance degraded after bulk loads), open a support ticket and ask for statistics and fragmentation analysis explicitly.

Summary Checklist#

• Full syncs scheduled, off-peak, on background processing nodes; manual full sync restricted
• readyForSync flag plus search restriction for controlled publication; restrictions re-enabled in the sync session
• relatedReferencesTypesMap extended for custom related types
• Part-of versus root type decided deliberately per catalog-aware reference; root types ordered dependency-first
• Prices and media evaluated for removal from sync entirely
• Threads, transactions, ordered relations, marking, and attribute list tuned in that order
• Sync dump CSV and DEBUG logger procedure documented in the runbook
• Repair scripts versioned and promoted with the release

Sync speed is architecture, not magic: the fastest synchronization is the one with the least to do.