Skip to main content

Network Status

CURRENT STATUS

Comcast collector circuits serving ICN K12 customers went down around 1:45 PM. We show they have recovered. Please contact our NOC if you continue to have connectivity problems. 312.814.3648 Option 2.

 

Announcement

Cisco Identifies Firewall Vulnerability:

https://sec.cloudapps.cisco.com/security/center/resources/asa_ftd_attacks_event_response

 

 

April 29 Outage - Comcast Summary of Event

Customer: STATE OF ILLINOIS

Comcast Root Cause Analysis Report

Incident Synopsis: On April 29, 2024, at 09:10 AM, Comcast was investigating the service outage for the STATE OF ILLINOIS, Comcast monitoring system logged the service down at 08:07 AM, on April 29, 2024.

Comcast engineer requested a same day dispatch to troubleshoot a the headend.

09:53 AM: An update was provided and no Estimated Time for Repair (ETR) was given. The Local Contact (LCON) was informed of the dispatch.

10:08 AM: Bridge was set up and in progress, investigation ongoing. Multiple tickets opened for the service issue linked under parent Service Incident (SI) ticket, SI066047234.

11:07 AM: Headend team troubleshooting verified no Receiver signal (Rx) from the location to the headend site.

11:12 AM: Strategic Account Operations Manager (SAOM) reached out to the customer and advised no ETR, investigation ongoing.

11:32 AM: Headend team verified Rx signals received from other nodes.

11:56 AM: Headend teams continued to troubleshoot. An Optical Time Domain Reflectometer (OTDR) test was conducted. OTDR test found optical light signal loss at approximately thirty-two thousand seven hundred feet out in the Comcast outside network plant. Investigation ongoing to pinpoint cause, no ETR.

12:23 PM: Dispatch request updated to add fiber repair.

12:55 PM: Fiber team engaged, enroute to the site for further investigation.

01:46 PM: Fiber team on site, troubleshot further, no ETR. Further investigation located fiber damage at 100 W Randolph St., Chicago, IL. 60601. Construction company performing renovation in the lower level damaged the underground Comcast fiber feed. Assessment and repairs underway, no ETR, updates to follow.

 02:51 PM: Customer called in for update on SI ticket, SI066046919. Comcast advised that ticket was resolved the outage is being worked under parent ticket, SI066047234.

03:05 PM: Headend and fiber teams collaborated to roll fibers, no ETR.

05:05 PM: Fiber teams verified light signal however, the routers remained offline. Troubleshooting ongoing, no ETR.

06:16 PM: Remediation efforts continued, links started to recover.

07:35 PM: Remediation efforts identified fiber optics for reroute, no ETR.

09:21 PM Fiber remediation efforts successful after spare fiber swap.

10:24 PM Headend and fiber teams declared hands off. A Scheduled Maintenance

Root Cause: Construction company performing renovation work at The Thompson Center, 100 W. Randolph St in the lower level damaged the Comcast fiber feed.

Comcast has installed two redundant fiber paths to the Illinois Century Network Chicago POP. Redundant hardware installation pending. Once complete Comcast will have service redundancy eliminating single point of failure caused by a fiber cut.

April 29 Outage (ICN Reporting)

8:35 PM: We confirm that all circuits have restored except for about 6 circuits which Comcast continues to work on. This will be our last update tonight unless there are new developments. Comcast will provide a summary of and cause of the outage and we will share that as soon as it is available. 

7:15 PM: We are seeing most sites restored. Some sites remain down and we continue to work with Comcast for full restoration. 

3:44 PM: Comcast is working to reroute ICN connections to new fiber path. That work has started. Still no ETA.

2:22 PM: Comcast believes they have identified a fiber cut and working to locate where.

12:15 PM: Comcast reporting possible fiber cut. To be verified. All comcast services to ICN Chicago POP facility are down. 

Comcast engineers are troubleshooting their hardware at our ICN Chicago POP facility.

8:30 AM: Comcast is reporting an outage on their network which is impacting multiple K12 Schools in the Chicago area. Engineers are working to resolve the issue. 

PLANNED MAINTENANCE

The Illinois Century Network (ICN) has scheduled network maintenance for February 23, 2024, starting at 11PM and ending by 1AM. Customers connecting to the ICN K12 Broadband Network may experience a brief interruption of service during the maintenance window.

Should you experience any problems with your ICN service after this maintenance, please contact our Network Operations Center (NOC) at 312.814.3648 Option 2 or at doit.icn.noc@illinois.gov.

Currently having Network Trouble?

The Network Operations Center (NOC) is our 24/7 support center. NOC staff monitor the network backbone, customer connections, respond to alarms and customer calls. The NOC is your first point of contact for reporting network problems and initiating technical support for your ICN connection.

217-524-3648Option 2 (Springfield)
312-814-3648 - Option 2 (Chicago)

DoIT.ICN.NOC@illinois.gov