Skip to main content

Possible online services disruption due to Internet related outage

A worldwide technology outage is causing disruption to some State of Illinois online systems.  We are aware of this issue and are diligently working on restoration.

Network Status

Current Status

No known network issues.

 

June 27, 2024 NIU/iFiber Outage

ICN customers with NIU/iFiber last mile circuits experienced an outage starting aound noon and lasting until approximately 2:40PM expereiencing an interuption of service due to a fiber cut. Fiber has been repaired and all sites are showing back up. To report a problem, please call the ICN NOC at 312.814.3648 Option 2.

 

June 12, 2024 Comcast Outage

June 12, 8:49PM - Comcast last mile circuits are restored. We will work with Comcast for an understanding of the cause of the outage which will be posted here as soon as availabe. 

Comcast reports repairs are in process. 

Comcast is reporting a fiber cut in the Chicago area. 

June 12, 7:00 PM - Reports of Comcast outage impacting K12 schools with Comcast last mile circuits. Will keep this site updated. 

 

May 22, 2024 Comcast Outage

May 22, 2024 Comcast collector circuits serving ICN K12 customers went down around 1:45 PM. Service restored by 2:15 PM. Please contact our NOC if you continue to have connectivity problems. 312.814.3648 Option 2.

 

Announcement

Cisco Identifies Firewall Vulnerability:

https://sec.cloudapps.cisco.com/security/center/resources/asa_ftd_attacks_event_response

 

 

April 29 Outage - Comcast Summary of Event

Customer: STATE OF ILLINOIS

Comcast Root Cause Analysis Report

Incident Synopsis: On April 29, 2024, at 09:10 AM, Comcast was investigating the service outage for the STATE OF ILLINOIS, Comcast monitoring system logged the service down at 08:07 AM, on April 29, 2024.

Comcast engineer requested a same day dispatch to troubleshoot a the headend.

09:53 AM: An update was provided and no Estimated Time for Repair (ETR) was given. The Local Contact (LCON) was informed of the dispatch.

10:08 AM: Bridge was set up and in progress, investigation ongoing. Multiple tickets opened for the service issue linked under parent Service Incident (SI) ticket, SI066047234.

11:07 AM: Headend team troubleshooting verified no Receiver signal (Rx) from the location to the headend site.

11:12 AM: Strategic Account Operations Manager (SAOM) reached out to the customer and advised no ETR, investigation ongoing.

11:32 AM: Headend team verified Rx signals received from other nodes.

11:56 AM: Headend teams continued to troubleshoot. An Optical Time Domain Reflectometer (OTDR) test was conducted. OTDR test found optical light signal loss at approximately thirty-two thousand seven hundred feet out in the Comcast outside network plant. Investigation ongoing to pinpoint cause, no ETR.

12:23 PM: Dispatch request updated to add fiber repair.

12:55 PM: Fiber team engaged, enroute to the site for further investigation.

01:46 PM: Fiber team on site, troubleshot further, no ETR. Further investigation located fiber damage at 100 W Randolph St., Chicago, IL. 60601. Construction company performing renovation in the lower level damaged the underground Comcast fiber feed. Assessment and repairs underway, no ETR, updates to follow.

 02:51 PM: Customer called in for update on SI ticket, SI066046919. Comcast advised that ticket was resolved the outage is being worked under parent ticket, SI066047234.

03:05 PM: Headend and fiber teams collaborated to roll fibers, no ETR.

05:05 PM: Fiber teams verified light signal however, the routers remained offline. Troubleshooting ongoing, no ETR.

06:16 PM: Remediation efforts continued, links started to recover.

07:35 PM: Remediation efforts identified fiber optics for reroute, no ETR.

09:21 PM Fiber remediation efforts successful after spare fiber swap.

10:24 PM Headend and fiber teams declared hands off. A Scheduled Maintenance

Root Cause: Construction company performing renovation work at The Thompson Center, 100 W. Randolph St in the lower level damaged the Comcast fiber feed.

Comcast has installed two redundant fiber paths to the Illinois Century Network Chicago POP. Redundant hardware installation pending. Once complete Comcast will have service redundancy eliminating single point of failure caused by a fiber cut.

April 29 Outage (ICN Reporting)

8:35 PM: We confirm that all circuits have restored except for about 6 circuits which Comcast continues to work on. This will be our last update tonight unless there are new developments. Comcast will provide a summary of and cause of the outage and we will share that as soon as it is available. 

7:15 PM: We are seeing most sites restored. Some sites remain down and we continue to work with Comcast for full restoration. 

3:44 PM: Comcast is working to reroute ICN connections to new fiber path. That work has started. Still no ETA.

2:22 PM: Comcast believes they have identified a fiber cut and working to locate where.

12:15 PM: Comcast reporting possible fiber cut. To be verified. All comcast services to ICN Chicago POP facility are down. 

Comcast engineers are troubleshooting their hardware at our ICN Chicago POP facility.

8:30 AM: Comcast is reporting an outage on their network which is impacting multiple K12 Schools in the Chicago area. Engineers are working to resolve the issue. 

PLANNED MAINTENANCE

The Illinois Century Network (ICN) has scheduled network maintenance for February 23, 2024, starting at 11PM and ending by 1AM. Customers connecting to the ICN K12 Broadband Network may experience a brief interruption of service during the maintenance window.

Should you experience any problems with your ICN service after this maintenance, please contact our Network Operations Center (NOC) at 312.814.3648 Option 2 or at doit.icn.noc@illinois.gov.

Currently having Network Trouble?

The Network Operations Center (NOC) is our 24/7 support center. NOC staff monitor the network backbone, customer connections, respond to alarms and customer calls. The NOC is your first point of contact for reporting network problems and initiating technical support for your ICN connection.

217-524-3648Option 2 (Springfield)
312-814-3648 - Option 2 (Chicago)

DoIT.ICN.NOC@illinois.gov