The following information provides more detail on the recent IT outages due to infrastructure maintenance or performance issues. We apologize for the inconvenience of these outages and are working diligently to avoid further disruption.
Campus Firewalls (August 10)
On Wednesday, August 10, 2022, we experienced an issue with our campus firewalls which impacted services for several hours. We engaged the vendor to help diagnose and repair the issue and have conducted an in-depth root cause analysis of this outage.
Load Balancing Equipment (August 11, 16, 23)
There is a pair of devices that help balance network traffic to many of SMU’s servers. This pair operates in high availability mode, meaning one device can go offline without impacting services—traffic is simply routed through the other device in the pair until the load balancing device is brought back online. OIT scheduled an upgrade to the devices on August 11. We expected no outage or impact to services during the upgrade, as all traffic should have been handled without issue by the other device in the pair. Unfortunately, an error in the configuration impacted both devices. The maintenance was quickly stopped and service restored.
The upgrade was rescheduled for the subsequent weekend (8/13-14) after reviewing the configuration to ensure a successful maintenance. There was a brief, but unexpected, outage around 7 a.m. on Sunday, which our team quickly resolved. The upgrade was completed, and it appeared the devices were operating normally.
We encountered another issue with these devices on August 16, resulting in further diagnosis with our vendor. After implementing the fix, the devices were running successfully and appeared stable—until August 23.
The issue causing the latest outage on August 23 has been addressed, and we are continuing to work with the vendor and monitor the system.
Wireless Network (August 17)
In order to support newer wireless access points recently installed across campus, we performed a routine upgrade of the wireless controllers. After receiving reports of intermittent wireless issues in multiple buildings, the upgrade was rolled back.
Spam Filter Appliance (August 18)
On August 18, we experienced an issue with our spam filtering appliance that resulted in delayed delivery of external emails. Once the service was repaired, all mail queued in the system was delivered.
