Are SaaS upgrades often so difficult and are errors irreversible?

  • 4 September 2021
  • 6 replies
  • 217 views

Userlevel 3
Badge +1

We’re trying to do our first major version upgrade since going live.  I’ve spent dozens of hours testing in the sandbox and the whole company is prepped (and excited!).

Then on Monday, the wrong version was ‘upgraded’.  No worries, mistakes happen.  But on Tuesday our partner was told there is now no upgrade path.  And Acumatica’s plan is to upgrade to a release that’s coming out next week and is 6 (minor) builds ahead of the sandbox we’ve been testing in.

 

Am I wrong in thinking this is dangerous plan?  I’m down for whatever my partner says is safe, but I’m more puzzled (and getting frustrated and concerned) with the ‘no upgrade path’ and the inability for Acumatica to fix it’s error.  What happened to all the benefits of SaaS, with our site super backed up and recoverable?  Can a sys admin really push a button and those backups be gone? 

 

If that is the case, then how come a non-recoverable upgrade is not confirmed before they happen?  Our partner or someone at our company should have received a message saying “This is a non-recoverable action.  Are you sure you want to upgrade to version ‘n’?”  That didn’t happen.

 

And there is nothing in the 20.121.0004 release notes stating there is no current upgrade path for major releases.  That’s 5 weeks before one appears.  Now, we’re only going to be stalled ~3 weeks, which can feel like a lifetime, but is recoverable.  Another company could be hit hard by this, say if they have a customization or need API calls that just can’t be made to work with a build that can’t be rolled back and they trusted Acumatica to truly back up their data (i.e. made no local snapshots with the correct build). 

I hope this reaches the right eyes and the policies around these issues get taken a look at.  

 

(I’m assuming SQL tables and DACs change in a way that make downgrades very difficult. And assuming the backups before the ‘upgrade’ are gone.  If I’m wrong, that’s good news for everybody, but then why didn’t we just get rolled back and upgraded to the correct version? I’d rather just be wrong too, so please everyone, correct what I’m not seeing. Thanks)


6 replies

Userlevel 5
Badge +2

I’m making up builds here, but were you on something like 19.111.0038 and the plan was to upgrade to something like 20.118.0007, so you did all of your testing on 20.118.0007, but the live upgrade took you to 20.121.0004 instead of 20.118.0007?

Then, was there some specific thing that worked for you in your testing on 20.118.0007, but it doesn’t work in 20.121.0004? And, since you can’t downgrade, you now have to wait for the next build to be released in order to get a fix for whatever it is that doesn’t work?

If that’s the case, you should have been able to restore the 19.111.0038 backup that was taken before the upgrade and then upgrade to 20.118.0007 as originally planned, but are you saying that no backup was taken? I find that hard to believe, but mistakes are possible.

Userlevel 3
Badge +1

Sorry Tim, should have been a bit more specific.  Our Live site was on 2020 R1 build 20.115.0039 (can’t remember exactly, but can find out as we’ve got a local instance still running with that same version), and the plan was to go to 2021 R1 build 21.108.0032, the version of the sandbox.

The mistaken upgrade took us to 2020 R1 Build 20.121.0004.  From here there is apparently no way possible to upgrade to 2021 R1 until a new build of 2021 R1 is released.  I don’t know why it is not possible to restore the 2020 R1 20.115.0039 backup.

 

Again, our situation is not so bad.  We take regular snapshots that are backed up locally.  2-3 weeks of slowed productivity; expensive, but fine.  What I’m trying to bring up is that situation where snapshots are not taken regularly and a upgrade build is incompatible, for whatever reason, with a customer’s instance. 

 

If you’re saying a snapshot is taken before an upgrade and that snapshot is still on Acumatica’s servers and we didn’t get rolled back because we didn’t raise enough hell, then okay, it’s recoverable.  Disappointing, but my theoretical customer would be okay.

Userlevel 5
Badge +2

Ah, interesting. So you asked Acumatica to upgrade you to 21.108.0032, but they accidentally upgraded you to 20.121.0004? And, even though 21.108.0032 is a higher build number, 20.121.0004 is “newer” so maybe going from 20.121.0004 to 21.108.0032 would be going “backwards” and you can’t go backwards because of they way that the upgrade process works on the technical side of things?

On top of that, they weren’t willing to restore from the backup that was taken before the upgrade? I’d be shocked if they didn’t take a backup before the the upgrade. Maybe they just don’t want to restore since you haven’t made enough noise about the issues that you’re having (which we’re doing now :grinning: )?

What are the issues that you’re having? What does “slowed productivity” mean? Is it related to a certain business process, to a customization, or maybe to an integration to a 3rd party ISV product?

Userlevel 3
Badge +1

Thanks for responding Tim!

 

I On top of that, they weren’t willing to restore from the backup that was taken before the upgrade? I’d be shocked if they didn’t take a backup before the the upgrade. Maybe they just don’t want to restore since you haven’t made enough noise about the issues that you’re having (which we’re doing now :grinning: )?

I don’t want to put all of this on Acumatica.  We currently don’t have a level of support that allows us to create cases directly, so everything goes through our partner (who we really like and I don’t want to put this on them at all either).  Whenever communication starts going through a 3rd party it’s like that old game of ‘telephone’..stuff is destined to be mis-communicated.   So I don’t know how hard it was pushed for them to restore us to 20.115.0039, then do the upgrade to 2021R1 21.108.0032 as planned.  It’s also possible the initial error was in the upgrade request from our partner.  2020R1 and 2021R1; it’s easy to see how that can happen.

 

Having had a little time to think about it I think I would have liked two things done differently by Acumatica: 

 

  1.  Be clearer that certain builds will block the upgrade path for the foreseeable future.  Our partner has done 300+ upgrades for clients and had never heard of this situation. 
  2. Obtain some type of confirmation for an upgrade.  We upgraded on a Monday night and Tuesday was a business day.  A simple “Are you sure you want to upgrade to version xx.xxx.xxx” seems like a no-brainer to me.
  3. (And maybe) just took the initiative to roll us back and do the upgrade if it was possible without us having to make a stink.

What are the issues that you’re having? What does “slowed productivity” mean? Is it related to a certain business process, to a customization, or maybe to an integration to a 3rd party ISV product?

 

We have a handful of outstanding issues/bugs/funky behavior with our instance, some of which I’ve posted about here, that have currently stumped myself, our partner, and Acumatica support.  On our partner’s recommendation we decided that we should do the upgrade before we try to tackle them again.  Some they are hoping just go away.  We have one nasty bugger that works in all of our test environments, but doesn’t in Live.  Some of them are standing in the way of fully implementing pretty major features, like Cases.

 

A lot of this is do to our size as a company, with absolutely no technical ‘department’, and the time we have to test.  I don’t know how common it is to still be doing setup and ironing out bugs 9 months after going live, but that’s been our experience.  We’re not just implementing Acumatica, but also making that transition from mom-and-pop small, to legitimately “small business” and our internal processes are sometimes getting hauled over as we do.  Trying to learn, test, implement, and troubleshoot Acumatica, while doing our normal non-software jobs, has been a challenge. 

 

So if I don’t say it enough, thank you Tim and everyone else who posts here or contributes.  We can’t be the only ‘small business’ Acumatica customer who benefits!

Userlevel 5
Badge +2

We can’t be the only ‘small business’ Acumatica customer who benefits!

 

I totally agree. This is not a finger-pointing exercise. This is an exercise in extracting lessons learned so that you don’t have to make the same mistakes a 2nd time. By posting here, hopefully others don’t have to make the same mistakes the 1st time.

I also agree with your partner about doing the next upgrade before wasting time tracking down bugs that may have been fixed, especially if you have some workarounds for things in the meantime.

Maybe, before you do the next upgrade, you can get a commitment from your partner and Acumatica to give you a rollback window. Something like, “the upgrade will be done between 6pm and 10pm Pacific Time on Monday, Noah has until 12pm Pacific Time on Tuesday to decide whether the upgrade went poorly and we need to restore from a backup.” That way you’d be on their radar. Acumatica is growing fast and they run pretty lean so I’m sure their upgrade team is stacked with a huge workload. I’m sure there is some kind of lead time on restoring a backup. That’s why it would be good to have a contingency plan ahead of time.

Also, note that there is a fee for restoring from a backup the last time I checked. I’m not sure if it’s applicable in an upgrade situation, but that would be something else worth investigating now before you go into the next upgrade.

 

Userlevel 3
Badge +1

I apologize to anyone who was following this thread that it took me so long to post an update.  As it was still going on I did delete a response I had written as I wanted to get some perspective and make sure I had my facts 100% sorted 

 

As always Tim, thank you for the thoughtful reply.  You’re idea about ensuring a commitment from Acumatica and our Partner about a rollback window for the upgrade process is a good one.  I’m not sure we’ll have enough sway, but it’s worth a shot.

 

We were correct in our assumption that the ‘upgrade path’ situation being something like a series of scripts that translates the database/DACs/codebase from one version to the next.  What happened with us was someone with the right qualifications at Acumatica finally took a look and they offered to do it those steps manually for free do to the situation.

 

I think in hindsight what felt so unacceptable about the situation was how long we were in the dark (almost 4 weeks) not knowing what the status of our site was going to be.  We even had to consider that Acumatica had hit such a snag and were going to just drop us as we are such a small customer.   If we just had a few minutes of the qualified tech’s time to explain what the issue was and a rough severity estimation, a good 85% of the frustration on our part would have dissipated.

 

It’s tough to say we’re all happy now, since we’ve discovered a bug with 2021R1 and entering Ship-To addresses on the Portal and our customers have been unable to place orders for over a month.  That case has been ‘elevated’ for 4 weeks now.  At least that’s some information, but customers not being able to place orders seems way more severe than the development stagnation I was alluding to before the upgrade was successful (we’re back to email and manual order entry).

 

I will go ahead and post one of the paragraphs I wrote, but never posted, during this thing, for two reasons.  One, if anyone at Acumatica reads it I hope it can be helpful and two, I’m guessing there are other ‘micro-businesses’ like us, with pretty much zero tech department, that can relate:

 

We totally understand the Acumatica support team is probably
overloaded.  We are and it seems our partner is as well.  These are
what I call ‘good problems’!  It could be the opposite and none of us
have enough business!  We hate having to tell customers something is
back ordered for several weeks, so we really do understand and
sympathize and I don’t mean this to sound too harsh...but, in this
relationship, one party approached the other claiming ‘if you sign
this contract, we are going to ease a lot of that burden”.  Well we
did, and some of our frustration is how much the onus has fallen on us
to get the other party’s software working and/or set policies and
workflows to implement those initial claims.  

 

Keep in mind we’re still very pro-Acumatica at our shop and everyone working with it is trying to convince our CEO to stick with it!  It’s so clear how much power there is to customize it.  I’m pretty sure I could not recommended it to another business like ours at this stage however.  Hopefully one day!

Reply


About Acumatica ERP system
Acumatica Cloud ERP provides the best business management solution for transforming your company to thrive in the new digital economy. Built on a future-proof platform with open architecture for rapid integrations, scalability, and ease of use, Acumatica delivers unparalleled value to small and midmarket organizations. Connected Business. Delivered.
© 2008 — 2022  Acumatica, Inc. All rights reserved