5/1/24 - Synopsis of recent system issue and resulting outage (message from Jas)

5/1/24 - Synopsis of recent system issue and resulting outage (message from Jas)



I want to personally apologize on behalf of me and my team for the inconvenience and hardship resulting from the outage and the missing data. 


First time in 15 years our systems were unavailable for just over 24 hours. On Sunday April 21 we experienced a hardware failure in the middle of the night. We worked round the clock on restoring the systems and were back online Monday morning. Due to unforeseen issues the restored system was missing 6 days of data from April 15 - Apr 20.


Since the early days in 2008, we have maintained all our systems in a high security data center using hardware we own and manage because of the sensitive nature of the data we took the responsibility of protecting. We survived all the security tests while providing an always-on system. We routinely upgraded our hardware and software to keep up with the cyber threats while enhancing the software capabilities. Unfortunately, a critical piece of hardware failed and took the system down.  At this point, we have re-organized our systems and data so this never happens again.


Some important facts I wanted to share. 

1. It was a critical hardware failure

2. It was NOT a security incident and no data was compromised

3. We are missing data that was entered between Apr 15 - Apr 20, 2024

4. We are continuing data recovery procedures to rebuild the missing data

5. All insurance claims were sent so it will not affect payments

6. All credit card transactions were processed but missing a record of it in our database

7. All uploaded documents are still there

8. ERA records are coming in and are in process of being loaded into the system


The last week has been very difficult for everyone. We organized ourselves into teams working in parallel. While the support team continued to help customers with the aftermath of the outage, the engineers were working on restoring the systems and the developers worked on software changes to rebuild missing data while the data recovery company is still working on the failed hardware.


In the meantime we are also implementing a new system for fault tolerance. Setting up a mirrored system that will eliminate the need to restore from backup, it will be a continuous backup.


We value the trust you have put on our team in protecting your data with our technology expertise so you can focus on your clients.


I want to assure you that we are doing everything to ensure we are never in this situation again.


Jaswant Pujari

Founder/CEO

Therasoft Inc