Skip to main content

Product change - Changes to how Connect import job queues work

Last changed: 2023-07-06

An internal change to Norce [Storm] functionality that might change the behaviour if the system integrations.

Background​

Importing data using connect is designed to be a process as effective possible. The design of the api and service has focused on throughput only, letting all jobs through as fast as possible. This meant that all the dispatcher of worker processes was designed to use them as soon as they are free again for next job, with no regard to the context of the job.

The "job starter" has the responsibility to trigger new jobs as soon as a worker is free.

This design has some flaws. Even though we have managed a high throughput, where large loads of products, prices or onhand lines passes through at an incredible speed, the actual efficiency has been less successful in certain situations.

  • Higher risk of data locks and deadlocks, making some jobs being delayed, and some lines not updated at all.
  • Increased risk of data integrity problems. When many jobs work in parallel with the same data sets some data that is older might override newer updates, or new products haven't had the time to being created when the next job tries to update the catalog.

In practice, these kind of problems sometimes makes for an experience of a low efficiency of the integrations, because of the need for re-runs and post-handling of data, and therefore we have decided on a new approach.

The Change​

With this change, Connect set up the queue so that all import jobs from the same client and import type queues together and processes them in order that they came in. The next in queue is not started until the one before is done. We created this by adding a new step before the Job starter from earlier.

The "Job starter" is the same as above, which triggers the process of new jobs as soon as a worker is free. But in this case, there is an order to the jobs the starter gets to process.

This change will solve several of the problems above, but may also come with issues to think about in your solutions.

Pros:

  • Each data entity will be updated in order based on when the jobs come in.
  • Less risk of data integrity errors, like trying to update a product not created
  • Less risk of long queue times because of a high load of jobs from someone else in the multi-tenant environment

Cons:

  • The window of time of processing imports can be longer. Mitigate this by aiming for more delta updates rather than full. The goal of the change is to make data more reliable so it should be possible to trust more to delta updates.

An import type if a specific endpoint in connect, 'ImportProducts', 'ImportOnhands', 'ImportPricelist' etc.

Timeouts and exceptions​

TODO: Do we have any times? Job is canceled after...

  • Use delta as much as possible and full only when needed to have a fallback (this change should make it possible to have them even less often, maybe weekly instead of daily, or bi-weekly instead of weekly)
  • Aim for data with good quality, that does not have missing critical information that may stop the batch processes and force retries. This will make the job take much longer. Examples of critical data is
    • Missing manufacturer, type or status on product (when creating)
    • Wrong or missing price list
    • Wrong or missing warehouse or location code
  • Plan for dependencies between import types. For example, if there is full imports of prices or inventory, make sure to schedule them when the full product import should be finished.

Other practices: See the system integration patterns here.

Performance data​

Here is some performance information, measured as lines/items processed per minute (LPM). Smaller jobs are jobs with less than 1000 items.

Data sample from June, July 2023

Old Prod environment​

Import typeSmaller jobs <1000 (LPM)Larger jobs (LPM)Note
ImportProducts2002000
ImportSkuPriceList40030000
ImportOnhand30050000
ImportCompetitorSku-20000
ImportRelation-20000
SupplierImportProduct--Not logged
ImportPriceList--
ImportCustomer40010000
ImportCompany2001000

Azure environment​

Import typeSmaller jobs <1000 (LPM)Larger jobs (LPM)Note
ImportProducts150700
ImportSkuPriceList3009000
ImportOnhand40010000
ImportCompetitorSku--
ImportRelation6002000
SupplierImportProduct--No data logged
ImportPriceList6001500
ImportCustomer--
ImportCompany-2000