CDG/QUANTUM GATEWAY DOWN: 2009-09-28 Monday

brettbrett FoxyCart Team
in Important News edited September 2009
Hello all.
Quantum Gateway started experiencing issues for FoxyCart merchants at (or soon after) 11:30:53 PDT today. No changes were made to the FoxyCart production servers, so at this point we believe the issue to be on Quantum's side of the fence.

We will be notifying all FoxyCart users who are using Quantum via email shortly, but please check this thread for updates as we get them.
Comments
  • looks like it's back up and running. can anyone else confirm this?
  • brettbrett FoxyCart Team
    Here's the status as of 9:40pm PDT:

    AFFECTED USERS:
    FoxyCart stores using Quantum Gateway.

    WHAT HAPPENED:
    At 11:31am PDT today (September 28, 2009), FoxyCart started getting SSL errors while contacting Quantum Gateway's servers. This caused all transactions submitted to error with a blank error message, which was presented to the customer. These errors are also logged in the store's FoxyCart admin error log.

    Once alerted to the issue, the FoxyCart team started diagnosing and troubleshooting. After confirming that nothing had changed on the FoxyCart production servers we contacted CDG/Quantum. We received responses shortly thereafter, and after some back and forth CDG/Quantum was able to restore service to FoxyCart users around 8:00pm PDT.

    At this point we don't know the root cause of the problem, but we do know how it happened. CDG/Quantum, as a gateway, must constantly upgrade their production servers to stay as secure as possible. They pushed out an upgrade this morning, after testing according to their own internal protocols. We believe that this same process has happened many, many times without issue, but this time, when their main system was being upgraded, the FoxyCart system made a request. Because the primary Quantum system was down, the load balancer on their end sent the request to their backup system. Unfortunately, their backup system and FoxyCart were unable to establish a secure connection. More unfortunately, since FoxyCart had now been routed to their backup system by their load balancer, FoxyCart _continued_ to be sent to their backup system even after their primary system was back online.

    While we still do not know the exact cause of the connection problems between their backup system and our own servers, Quantum was able to remedy the situation for the time being, thus restoring functionality. Some of our team will be working with them tomorrow to ensure this does not happen in the future.

    WHAT YOU SHOULD DO:
    Check your error log in your FoxyCart admin. If you do see any errors, you should _also_ see the customer's information. You could, if desired, contact your customers to let them know what happened, and that your stores are functional now. (Perhaps a coupon code would be worthwhile as well. Just a thought.)

    WHAT WE'RE DOING TO PREVENT THIS IN THE FUTURE:
    One of the biggest challenges with e-commerce is that it, out of necessity, involves _many_ external systems, be they shipping carriers, payment gateways, the cardholder's issuing bank, and etc. While we have robust monitoring of our own systems, we do not monitor the dozens of external systems that FoxyCart interfaces with. As a result of this error, however, we are exploring options to monitor gateway connectivity. While this wouldn't prevent the problems, it could potentially decrease the time to resolution in the future.

    Also, as mentioned, our team will be working with CDQ/Quantum to ensure we don't have this problem in the future.

    We take all issues like this _very_ seriously. Please let us know if you have any questions or concerns.
Sign In or Register to comment.