Downtime

Posted on 08 July 2011 by – Comments (15)

You may have noticed that we experienced some unexpected and unscheduled downtime this morning. We're really sorry about this and want to give you a full update on what happened.

During our scheduled maintenance period last night we pushed a major update to FreeAgent, upgrading our software platform to Rails 3/Ruby 1.9.2. This morning we realised there were some issues arising from this so we made the decision to roll back to the previous version of the software.

This took longer than we would have liked because we wanted to check that the issues discovered this morning were definitely resolved.

I just want to let everyone know that there's absolutely no data loss or corruption associated with this downtime, so there's nothing to worry about there.

I'm very sorry that we let you down this morning. We set ourselves high standards with respect to the service we provide, and we've obviously not met those today.

Over to you...

Dermot Daly, Fri July 08, 2011
It is precisely by providing this level of transparency, you will continue to have loyal customers. Stuff happens; Its how you react and respond that matters.
Kudos.
Robert Lo Bue, Fri July 08, 2011
100% agree with Dermot. So long as companies are transparent we are always willing to forgive a few hours downtime. Thanks!
Graham Cheale, Fri July 08, 2011
Completely agree with first comment, it has been inconvienient - but we certainly appreciate your open approach.
roan, Fri July 08, 2011
Thanks everyone. That kind of support and encouragement really means a lot to us.
Kit Barker, Fri July 08, 2011
Like Dermot said, stuff happens.

This is the first real downtime I've experienced since I started using FreeAgent about a year ago.

It shows the level of uptime we've come to expect and the usefulness of the product that being unavailable for a couple of hours seems like a disaster!

As long as this level of downtime feels like a disaster to me, you're doing a good job. It's only when it becomes expected that you're in trouble!
Jay McGuinness, Fri July 08, 2011
It just goes to show how good your uptime has been previously, as I'd forgotten that FAC was an online service until this morning.

As a pinned chrome app window it feels very much like a desktop app sometimes (in a good way ;o))
Chris Blunt, Fri July 08, 2011
Agreed, your openness and honesty is greatly appreciated - thanks for the update.

You also deserve credit for upgrading FaC to Rails 3 - I know that's no easy task, but worth the effort for all the benefits v3 includes :)

Keep up the good work!
Liz Bell, Fri July 08, 2011
Completely agree with the other comments. I did try and log on during the downtime, but it wasn't urgent. To get an explanation via Facebook within a couple of hours is pretty amazing. Keep up the good work!
Jon Moss, Fri July 08, 2011
Roan, appreciate the post, your honesty and obvious commitment to your customers.

As with all other comments, I echo their thoughts and keep up the great work.
Philip Cunningham, Fri July 08, 2011
excellent communication via Twitter @freeagent Keep up the great work
Marcos Scriven, Fri July 08, 2011
A lot of your customers, I bet, are software engineers, and know this happens from time to time. It happens to even the biggest companies.
Sean Burlington, Sat July 09, 2011
Advance notice of downtime, testing of upgrade, rollback, and the magic words "no data loss"

Good work, good communication, and I'm a happy customer.
Dennis Howlett, Sat July 09, 2011
@roan - well done on transparency. What lessons can you pass on? Immediate qus: Do you believe there was a bit of complacency given FA has not needed to report significant unplanned downtime for some while? Did your team allow the pressure of getting out an update to interfere with testing thoroughness without realising it? Do testing suites need more thorough updating? Does Rails 3 introduce new vulnerabilities of which devs need be aware?

Lessons passed on are invaluable to others in my experience.
roan, Sat July 09, 2011
@Dennis: Olly will be posting technical details of what was involved with this release in the next few days.

I will say that we definitely weren't complacent though. Upgrading to Rails 3 represents a huge change to our application platform and has taken the team over 2 months of development and testing. We believed we'd tested this thoroughly but as with any change of this scale, it's always possible that something will slip through, as was the case here.

There are lessons to learn though, and we'll be making improvements for the future.
Dennis Howlett, Sat July 09, 2011
Nice one Roan. All goodness and looking forward to Olly's analysis.
(Comments closed)

Recent Posts

Archive

Categories

Twitter Love

  • Thanks to @freeagent I will never need to use the crappy HMRC payroll software come April.

    @lukeredpath

  • all sorted and ready for PAYE RTI thanks to @freeagent awesomeness.

    @davewardle

  • The latest reason why I continue to love @freeagent http://t.co/nfYqVJDfa4 HMRC chucks a stress out at businesses, FreeAgent bats it away!

    @myriammelot

  • Those folks @freeagent do it right. The confirmation mail includes important information: http://t.co/HCHIvgjpI0

    @dermdaly

  • @NickClement @freeagent its excellent for invoicing, very flexible & easy - esp billable time by task from timeslips-capture the revenue!

    @soscompass

Follow:

Escape spreadsheet hell with FreeAgent.

Try FreeAgent for Free

FreeAgent Online Accounting Software. Copyright © 2006–2013 by FreeAgent Central. All rights reserved. Location: