Question

Using timestamps as basis for concurrency control is a bad idea

R
Rank
Userlevel 6
Badge +5

I have been doing some work with Webhooks and Acumatica, and have been running into the problem where an update done via Webhook will cause concurrency issues with editing the same Sales Order via the Acumtica front end.

The most common error is: “Error: Another process has updated the 'SOOrder' record. Your changes will be lost.”

In researching this issue and its underlying cause, I came across this blog post:

https://asiablog.acumatica.com/2018/03/another-process-has-added-updated-deleted.html

It seems that Acumatica is using record timestamps a a way of managing concurrency. However, this seems like a bad idea. Especially in a web based world where many users can potentially update a record, it seems we would want to use the actual content being saved to determine if we have a concurrency issue or not. If there is no conflict, then we shouldn’t care whether one user is saving on top of another (or at least, we should let the user decide whether it’s a problem or not).

The most obvious example I’ve seen of this is when I will:
1) Open a sales order in acumatica
2) Save line items to sales order externally and save -via webhook code
3) modify sales order header in open  acumatica window and try to save…..conflict

There is no overlap or conflict with the data in this case. If one were to hash the individual fields of the three objects, the original sales order object, the new one saved via webhook with additional line items, and the final object with modfiications to the header (and the new sales order lines...since they are pulled in via the transactions section refresh), there is no inherent conflict here. The only conflict is with the timestamps.

Is anyone at Acumatica thinking about this issue? I would point to Git as a better model for managing concurrency than the one being employed here...

6 replies

Dmitrii Naumov
Rank
Userlevel 7
Badge +5

One thing that you are missing is dependent fields. 

E.g. if you edit lines, the change may need to be reflected in the header. For instance, when you edit amount of line, the header would reflect it in one of those fields that contains totals. 

Theoretically, one can define what fields depend on what fields and merge the changes. However, this is very hard to manage and introduces significant restrictions on the way the business logic should be written. If you add possibility of any code to be customized, that becomes completely impossible.

R
Rank
Userlevel 6
Badge +5

 @Dmitrii Naumov I’m not trying to claim that concurrent updates, or the implementation of good optimistic concurrency control, is easy. This is a hard problem.

However, there are some strategies that can reduce the pain quite a bit. For example, you could have a list of stock fields that have no downstream dependencies. For example, the “Description” field for the Sales Order probably shouldn’t conflict with a previous update that added a Sales Order line. If an update has occurred to a Sales Order line, followed by an update to the Description field, then no exception is raised. You could use some sort of bitmask to compare these on save.

The problem I have is that this optimistic concurrency control as currently implemented seems to hang exclusively on the timestamp field. I think this leads to a lot of issues, especially as you implement features like webhooks, where ultimately it is the user conflicting with his/her own changes. It is very annoying to fill in a bunch of fields and then have the system reject them, especially if you were the same user that added the conflicting change through a webhook integration.

Can the current concurrency management logic be changed through customization in Visual Studio?

 

M
Rank

Might I recommend a different solution?  Instead of counting on Acumatica to merge the records together, maybe there should be a way for the user to do it.  When an update record is flagged as written by another user, then pull the new data for that record and have the user cherry pick their values or the servers new values where there are differences.  This is similar to a merge in visual studio when the git repository was updated.  It is up to the user to decide which data is correct.  That seems better than asking the user to start over.

R
Rank
Userlevel 6
Badge +5

Hi @MikeWodarczyk -

Yes, in the case that there are actual conflicts between user updates, then that is a reasonable approach to the problem. My suggestion to compare individual field hashes would sometimes identify true conflicts between fields, and at that point a solution like you suggest would be one way to resolve the conflict. Comparing the hashes would also be a pretty efficient way to identify which field caused the conflict and to then prompt the user as you suggest.

The motivation for my original post was to point out that it’s far worse than that currently. If you build an external integration, it is far more likely that it’s the same user that is conflicting with their own changes. This occurs because of the simplistic nature of the concurrency control, where the only thing that is checked is the timestamp that as sent when the user loaded the page. if they issue an update to a record via webhook in the intervening period, then that update will conflict with any subsequent update done via the native website. The point is really that if you are going to allow external updates to the system via third party sources (which Acumatica is absolutely right in doing...this is the future!), then they need a better way to manage concurrency that doesn’t drive the users insane.

 

M
Rank

@rosenjon. I see now what you are getting at and indeed that is a problem!  We are going to be in a similar situation soon. This is good to know.

It sounds like the Acumatica native website tracks the record’s state over the time period of the user edits.  When it comes time to write the state to the database, it has been made stale by a webhook event on the same record. 

What Acumatica probably should be doing in the UI is tracking changes to the state over the duration of the user edit.  When the user clicks save, they should then checkout the record to get a fresh copy, apply the changes, and save it.  The save procedure would be done in 100 of milliseconds making it less likely to fail the timestamp check.  Using a retry technique, like Poly, the much less frequent timing conflicts could be handled.   

There would still be some scenarios that wouldn’t work. E.g. if the web hook deleted the record then the changes could never be applied.  Or if an account or inventory would go negative because of the user changes being applied on top of intervening web hook changes, then the user change would have to be rejected.  But I suspect that your scenario is much less fragile than that.

Another approach which was done in Sage and has caused us similar amounts of pain, is to lock the record while the user had it open in the UI.  The webhooks would fail, but the user could get their edits done.

I guess concurrency in financial systems is a tough nut to crack.

R
Rank
Userlevel 6
Badge +5

@MikeWodarczyk Indeed, as I mentioned above, this is a “hard problem” in computing. Made harder by the fact that Acumatica may update multiple data sources from a single screen, therefore cauisng unpredictable side effects if you try to “guarantee” that one change will not conflict with another. There are also the direct dependencies like order totals, etc that @Dmitrii Naumov mentions above.

Your reference to Polly is interesting: https://github.com/App-vNext/Polly

I have not used that framework before, but maybe the Acumatica Dev team will want to give it a look.

You also mention the Sage approach known as pessimistic concurrency control. This is ok in a thick client server environement. It works less well in a web environment where it can be more difficult to determine whether a record lock is even still valid (a.k.a this is a mess with web based environments). The result can be lots of locked records that no one can edit, which also drives the users crazy.

I appreciate where the Acumatica devs are coming from right now to some degree. If you open up concurrency control to the mob, you may create a situation where people will corrupt their own systems, and then they will be blamed for giving those people the gun with which to off themselves.

At the same time, if they are going to try to have updates via webhooks and external sources, there needs to be a better way to manage this. I think a good place to start would be having a better system for identifying what has changed and by whom. If the same user is updating a record (ie via webhook integration), then there is less of a problem. The user knows what they have changed, and therefore as long as the system can correctly show the current state of the system, there shouldn’t be many surprises.

I have managed this currently by using the javascript window.opener technique I mentioned in a previous post, where the user will submit a change via webhook and then we reload the screen. This ensures that the record has been updated, that the UI has all current updates, and that the timestamp has been properly updated so there is no conflict. The only downside is that this requires the page to refresh. Really the only problem is that for now it requires a page refresh. Maybe Acumatica devs could follow that line and figure out how to achieve the same thing without a page refresh. If they were able to store the user’s UI edits, then they could reapply them after the transparent reload, and THEN warn the user that the latest record has been loaded and highlight the current changes in red.

Something like that...

 

Reply


Terms & ConditionsCookie settings
About Acumatica ERP system
Acumatica Cloud ERP provides the best business management solution for transforming your company to thrive in the new digital economy. Built on a future-proof platform with open architecture for rapid integrations, scalability, and ease of use, Acumatica delivers unparalleled value to small and midmarket organizations. Connected Business. Delivered.
© 2008 — 2024  Acumatica, Inc. All rights reserved