Real-time and Scheduled Synchronization for Acumatica BigCommerce and Shopify Connectors

2 years ago
27 April 2021
1 reply
929 views

Userlevel 7

+3

smarenich
Acumatica Commerce Edition Team Lead
227 replies

This article will be focused on the specifics synchronization used in the Acumatica Commerce Edition for Shopify and BigCommerce connectors.

First, lets define the synchronization process - Synchronization of any entity in Acumatica Commerce edition happens in 2 steps: Preparation and Processing.

Preparation is a process when Acumatica gets information that any entity requires a synchronization (ex, new order has been created or product has been updated). When information is received, connector needs to create a record in the Sync History screen, logging that the record has been received and pending sync.

Processing is a process when Acumatica loads one single record and synchronizes it from one system to the other. As part of the processing, connector will perform some addition steps: getting record from both systems, fields mapping from source to the destination system, user defined mapping processing, error validations, duplicate validations, saving the record to the destination system, updating the Sync History to reflect the processing.

As you may notice, Processing may take significantly more time than Preparation. Moreover, Preparation may leverage batch processing to get multiple items at one, where processing has to process record by record. And finally Preparation and Processing can be executed in parallel - processing does of the record can be safely done, while preparation is adding new records to process.

Now let’s define different types on synchronization:

Scheduled Synchronization

The scheduled synchronization is the process where both Prepare and Process operations are executed by the schedule. For instance, prepare can be executed every 10 minutes and process every 5 minutes. Another case is where prepare and then process are executed only every night to perform all the needed operations out of the working hours. As Prepare and Process are scheduled independently, they may be performed in parallel and each of them will consume one tread or one core. Please note that Acumatica is designed in the way where one scheduler cannot be executed again until the previous execution is completed, that means scheduler will complete the operation before starting it again. This is needed to protect online users’ performance, as multiple schedulers may consume all possible server resources and user interface will be unresponsive.

It is also possible to enable multi-thread parallel Processing of the prepared records for Acumatica instances where resources and license allows it. Parallel processing is available only for Process Data and allows to use up to half of the available cores to process multiple entities in one given moment of time. The other half of the cores will be reserved for UI requests to ensure the best user experience. Parallel processing needs to be explicitly enabled in the web.config file.

Real-time Synchronization

Real-time synchronization is the process where Acumatica tries to Prepare Only or Prepare and Process as soon as change happened on one of the systems. For example if order has been created on the store, external system generates the web-hook notification that is sent to Acumatica as soon as order is placed.

As Acumatica may receive hundreds and even thousands of the notifications in a minute, the Acumatica does not process all the web hooks immediately, instead connector will record the web hook in the special system queue for the further processing. This is done to protect online users’ performance, as multiple notifications may consume all possible server resources and user interface will be unresponsive. Acumatica has a background processing thread that will slowly process all the incoming notifications. This processing thread cannot consume more than one core, to protect Acumatica server from overloading. As processing of multiple requests may consume some time, in some heavy cases, real-time synchronization may have delays based on the load of the server.

Another important point is that real-time synchronization has to modes: “Prepare Only” and “Prepare and Process”. The first mode will only update Sync History, where the Processing should rely on the scheduled processing. The second mode will not only update Sync History, but also will synchronize the updated entity immediately. It is extremely important to say that preparation and processing happen on the same background thread, so leads to significant slowness of real-time synchronization, as background process will not start the next sync, until the previous record is processed. In some unlikely cases real-time sync may not be real-time at all, as queue will be filled faster than processing happens. It is fully depending on the actual volume, but in case you have more than 100 synchronizations per hour, most probably Prepare Only mode with Scheduled Processing will be a better choice for the system performance and scalability.

It is important to remember about the limitations of real-time synchronization:

Import real-time synchronization is not reliable, as web hooks may be lost, disabled or deleted. In this case Acumatica will not receive the notification and corresponding sync will not be performed. It is highly recommended to schedule incremental night Preparation to be sure that all changes are reflected and synchronized.
In case of failure, real-time synchronization will not retry the sync after some time, as the background process will continue to process the queue further. It is highly recommended to schedule regular Processing to be sure that all failed records re-synchronized occasionally.
Import real-time synchronization requires your Acumatica to be available in the internet with public available URL (without any VPN) and valid SSL/TLS Encryption Certificate. This is needed as the third-party system need to send the message to Acumatica directly. If message cannot be delivered, then import-real time synchronization will not work.
In some cases, Acumatica or External System may generate multiple notifications for the same record. For Instance, when shipment is processed, Acumatica may generate multiple notifications updates to the sales order. To improve the performance and minimize duplicated actions, Acumatica Connector will take a 15-20 seconds delay in processing of the incoming notifications processing to ensure that all similar messages are processed at once.

Performance optimizations for the fastest processing

Recommended setting for the best performance will be different depends on the volume that requires processing.

Standard volume setup - where total number of synchronizations are below 500 per hour.
- Start real-time Synchronization for Sales Orders and Shipments with Prepare Only Mode.
  - Please note that Customers and Payments will be automatically synchronize together with order.
- Configure scheduler for the night fetch for all needed entities.
- Configure scheduler processing for all entities every hour, so prepared records can be synchronized sooner.
- Add more frequent Prepare and Process schedulers for the other entities that require more frequent processing.
Large volume setup – where total number of synchronizations are above 500 per hour.
- Stop real-time Synchronization for all the entities.
- Schedule Prepare for Sales Orders anв Shipments for every 10 minutes. This will help to leverage batch fetch, as batch processing is faster than individual push notifications.
- Configure scheduler for the night fetch for all needed entities.
- Configure scheduler processing for all entities every hour, so prepared records can be synchronized sooner.
- Add more frequent Prepare and Process schedulers for the other entities that require more frequent processing.
- Enable Parallel Processing.