I just wanted to convey a bit of additional information.

Tech President journalist Nancy Scola interviewed me, and wrote a great article about this incident.  She also dig a bit more digging into the actual cause of the outage.  She says, amongst other things:

That 'bunch of things,' says a PR rep for Packet Exchange, Salsa's ISP, traces back to a FiberLight, the Georgia-based fiber optic cable company that Packet Exchange relies upon for its Internet service in the Washington DC area. According to the rep, a construction accident yesterday at a Reston, Virginia data center ended up with a cut FiberLight cable, which cascaded down until Salsa Labs found itself without service. "Outages happen," said the rep. "Fibers get cut." 

We're making great progress on setting up a third alternative ISP.  It'll be up soon!

Read more (6 comments)

After yesterday's ISP outage, we want to ensure you all know what we're doing to ensure that we're not going to be affected by this type of issue again.

Despite the numerous safeguards we have in place, the failure we experienced exposed a rare, but clearly damaging flaw with our ISP providers.  As a result, we are expanding the number of ISPs that our systems are connected to, ensuring that no dual failure, which we experienced yesterday, will affect us like this again.  

In the end, these situations can be tricky, but rest assured that we are, and will continue to do anything necessary to expand our capacity, increase our availability, and continue to operate as the thought leader in the online Progressive movement.  

Cheers,
/Justin Nemmers
COO, Salsa Labs, Inc.

Read more (40 comments)

I'd like to publicly thank some specific people who helped support our customers and members throughout yesterday's ISP outage.

First, Andy, our lead Systems Administrator adeptly handled the vendors.  Matt's heroic dash to the data center allowed us to make sure we would be ready to hit the ground running once we regained connectivity.

Two, Leslie's entire member services team-- the people you talk to when you call us-- stayed late to field calls from you, and allowed us to keep the support lines open until after our ISP came back online. Specifically, Lin, Christopher and Brett were here until well after the fiber cut was repaired to make sure the queued blasts were progressing, and that new tickets were logged and responded to.  We even got some other Directors in the call rotation, so thank you, too, Geoff!  

I may be relatively new here at Salsa Labs, but there's one thing that I can say right now. I was nothing short of amazed by the hard work, dedication, and passion that each of our employees showed during this time.  It's their efforts that help make this such an amazing organization.

Onward,
/Justin Nemmers
COO, Salsa Labs, Inc.

Read more (145 comments)

I've had a lot of people asking me what "fiber cut" means. So here it is.

A fiber cut is a physical cut of a fiberoptic cable. Generally a fiber c happens when a work crew is digging with heavy machinery. Fiber is incredibly difficult to splice back together because it is glass, and requires a sterile environment to repair.

In order to fix it, a special fiberoptic cable repair truck literally has to park on top of the location of the cut and both ends need to be brought into the truck before you can begin the repair. It is very rare and takes hours to mend.

Read more (5 comments)

Hi all:

I know a lot, if not all of you, are curious about exactly what happened with the ISP outage. We'll be updating this blog all day today as we get more information from our ISP about what happened yesterday with the fiber cut. Keep checking for updates.

Thanks!
Leslie

Read more (655 comments)

At 3:46 pm EDT on Wednesday 19 May, Salsa Labs' primary Internet service provider suffered a cut of their fiber connection linking our data center to their network.  As a result, all Salsa Labs, Wired for Change and DemocracyInAction.org sites were down for approximately 6 1/2 hours.  

During this time, the Salsa Labs staff worked tirelessly with our various vendors to develop and implement a workaround to allow the flow of traffic through other networks while our primary provider continued to work on a repair.  Please follow our posts on this specially created blog to learn more about steps we will take to be sure this doesn't occur again.  

Salsa Labs apologizes for the disruption your organization has experienced as a result of this outage.  

If you have additional questions or comments, don't hesitate to give us a ring.  Our main number is 202.558.2807, and select the support option.

Regards,
Justin Nemmers
COO, Salsa Labs, Inc.

Read more