One of our core routers, core-1.msp was rebooted several times for emergency maintenance.  The outage resulted in loss of Internet connectivity for most Infinity customers.

The outage was not a success, and will require additional work.

A group of 4 ports went out on a key network switch -- one of the four ports was an uplink to a core router.   This resulted in lost connectivity for several co-located customers, hosted websites, and dialup isdn/modem access.

A technician was dispatched to move the four affected ethernet connections off of the four bad ports.   Connectivity was restored immediately after this at 2:38AM.

A server outage resulted in the complete loss of naming services.   This prevented DNS resolution, some mail services, and would have also caused authentication errors for DSL, ISDN, and Dialup customers, but would not have affected users already logged in.

We were able to bring the server back up without much trouble.   An emergency outage is likely this evening to replace the troubled server.   In that event, another outage notification will be posted.

XO maintenance to fiber mux resulted in approximately 10 minutes of down time of some customer T1 circuits, and some of our upstream Internet connectivity.

All services up and operational.

At 12:38PM Infinity noticed problems with our core network.

Initially the problem seemed to be a Denial of Service 
attack. The problem was later isolated to upstream 
connectivity provided by Time Warner Telecom.

Many thousands of packets per second were coming 
in on this interface which was severely degrading 
routing performance in our core network.

The Time Warner interface was shut down and 
all services were restored approximately 45 
minutes after the outage began.

Time Warner Telecom confirms that they are
experiencing a complete outage with their native
lan service in their Twin Cities market, and has
no ETA.

Infinity is currently operating without Time Warner
Telecom service until it is reported that they have
recovered from this outage.

Update: 10 Sep 2004, 2:55 P.M.:
 Time Warner Telecom reports that they have
 fully recovered from the problem.

 We are waiting on a report from a Time Warner
 technician, as to the nature of the outage, and
 what they did to resolve it, before we bring the
 circuit back online.

Time Warner Telecom connectivity was brought back
up on Monday 13Sep2004 at 10:20 A.M.

Building and many Eden Prairie regions lost power from 
NSP. Data center switch lost power from local UPS when 
delayed start on data center backup generator. 

Local UPS supplying power to switch was changed out 
and switch was back on line in a few minutes.

Disk problems on one of our NT webservers is causing 
multiple virtual-hosted websites to be down. There is 
currently no ETA. A technician is working on this issue
to be resolved as soon as possible.

Disk problems took the server down.  Web sites were 
moved to different server.

Several co-located servers became infected with 
W32.SQLExp.Worm and caused CPU utilization 
on our core router to reach a very high level. 
This severely degraded routing performance.

The offending servers have been disconnected 
from the network, and will not be reconnected 
until they have been cleaned and protected 
from further infections.

Furthermore, ports 1434/tcp and 1434/udp have 
been filtered from inbound and outbound traffic.

Hardware problems on one of our Unix webservers 
is causing multiple virtual-hosted websites to be 
down. There is presently no ETA.  A technician is 
working on this issue and it will be resolved as 
soon as possible. Disk mirroring in conjunction 
with file system journaling seems to be the culprit.

Web services were down for roughly 1.5 hours. An 
OS bug with running disk mirroring + fs journaling 
caused the /web file-system to become dirty. This 
caused a kernel panic which was bringing down 
the server. The server was brought down and its 
file-system was forced clean. Disk journaling is a 
feature that (to put simply) allows for faster 
reboots after crashes.

This feature has been disabled for now. The server 
had been running for 1.5 years without a reboot.
In the same amount of time, we 

Hardware problems on one of our Windows NT IIS 
webservers is causing multiple virtual-hosted 
websites to be down. There is presently no ETA. 
A technician is working on this issue and it will be 
resolved as soon as possible.

Hardware problems on one of our Windows NT IIS 
webservers is causing multiple virtual-hosted 
websites to be down. There is presently no ETA. 
A technician is working on this issue and it will be 
resolved as soon as possible.

Group outage at Onvoy.  An Onvoy->Qwest DS-3 is 
down. Qwest is working on repairing this DS-3, and 
it will be up as soon as possible.

All traffic is being routed over our Sprint connection. 
Network slowdowns are to be expected.

All circuits came back online at approximately 
1:56 P.M.

One of our Windows NT virtual-hosted web servers 
is having issues.  Several websites on this server 
are down. There is currently no ETA. Websites 
will be back online ASAP.

The server had locked up. Rebooted and 
everything is running fine again.

90% packet loss on Qwest America DS-3. Our 
BGP session to Qwest was flapping. The BGP 
connection was shut down by Infinity for the 
duration of the outage to prevent further BGP 
dampening. Dampening had taken affect and 
Infinity was dampened for approximately 15 
minutes. Once undampened, our Sprint 
connection took the full load and was at full 
utilization.

Qwest is unable to explain the outage.  It does 
not appear to be at the ATM layer on the 
Infinity side of the circuit.

One of our frame circuits lost connectivity to the 
frame network.  Customers with PVCs connecting 
to this circuit are down. Connectivity was lost at 
roughly 12:15PM. LMI was down.

Qwest technician found that a frame card in a 
switch at Qwest was locked up.  He reset the 
frame card and LMI came back up.  As of 1:15PM 
all PVCs were up. Qwest will most likely replace 
the frame card if the circuit does not remain stable.