It looks like you're new here. If you want to get involved, click one of these buttons!
Bugs & Feature Requests
Gateways, Merchant Accounts, Bank Accounts, Oh My!
FoxyCart is DOWN - 20090401 Status
edited April 2009
We are currently experiencing connectivity issues, as of April 1, 2009, 7:00 PST (Apr. 02, 02:00 Universal Time).
We will update this thread with info as it is available.
We are currently back up. According to our records (and our 3rd party monitoring service) we were down from 7:07pm-7:34pm PST, which is Apr. 02 02:07-02:34 universal time.
We are examining the cause of this downtime and we will post with details once they are available.
We have identified the cause of the downtime, and are currently taking steps to ensure that it doesn't happen again.
Our datafeed processor currently runs every minute, and attempts to feed every "unfed" transaction to stores that have a datafeed URL set. Earlier today a store turned on the datafeed for the first time and it attempted to feed the entire history of the store, going back over a year.
This huge amount of data, in addition to a lack of hard limits on transactions per datafeed, caused our primary server to go unresponsive.
What we're doing to prevent this from happening again:
v051 will bring "instant datafeeds", so most of these changes won't affect stores for that long, but for existing FoxyCart versions we are taking steps to limit the number of transactions that are attempted to be fed per datafeed. This means that if is exceptionally heavy traffic system-wide that your datafeed might come in delayed for more than one minute. In practice we don't expect this to be an issue, but it is now a possibility.
We are also lowering the limit of datafeed failures per store to 3, after which point the store's datafeed will be disabled.
Perhaps worth noting...
Amazon.com, SalesForce.com, and others (including Register.com, though perhaps for unrelated reasons) all
experienced downtime today
. Granted, it was the result of a massive DoS attack and not an issue with their datafeeds, but today seems to be an unlucky day for uptime. That's not an excuse, but it is interesting to note that even the biggest sites in the world go down. We'd like to think it was "preemptive sympathetic downtime", but perhaps not.
Please understand that we at FoxyCart take our uptime _very_ seriously. We were aware of the downtime even before we received our emergency notifications, and it was "all hands on deck" until connectivity was restored. We realize that even a small amount of downtime can impact sales and profitability, and we sincerely apologize for any loss of sales.
While we don't promise 100% uptime (because it's impossible), we do promise to come as close as we are able. If anybody has any concerns please don't hesitate to whisper us, or post on this thread.
We appreciate your understanding, and don't take lightly the trust you've placed in FoxyCart to handle your clients' ecommerce needs.
Thanks for being on top of this, clear communication, and a good response. It's great to see you figured out the problem and how to keep it from happening again in an hour.