Skip to Content

System News

Filtered by: System Notices | (Remove Filters)

  • Blue Waters Returned to Service

    Created by: kingda 2019-09-12 16:28:51 (Channels: scheduledoutages|systemnotices)

    Blue Waters returned to service at 4:00PM following today's scheduler maintenance. 

    Please email help+bw@ncsa.illinois.edu to report any issues.

  • Scheduled systems testing period on Thursday,¬†September 12th¬†from 10AM to 5PM Central

    Created by: gbauer 2019-09-12 09:51:55 (Channels: scheduledoutages|systemnotices)

    A scheduled systems testing period will take place on Thursday, September 12th from 10AM to 5PM Central,  necessitating a shutdown of the job scheduler. Compute nodes will not be available during the test period. Login nodes and file systems will remain accessible.

  • Blue Waters: NPCF Power Issue Update

    Created by: tbouvet 2019-07-06 03:45:32 (Channels: unscheduledoutages|systemnotices)

    Blue Waters: NPCF Power Issue 7/5/2019 3PM

    All Blue Waters Resources are available except for the compute nodes. Blue Waters Computes are being rebooted and all running jobs were lost. No RTS eta yet.

  • Blue Waters Returned to Service

    Created by: bbode 2019-07-05 17:57:32 (Channels: unscheduledoutages|systemnotices)

    Blue Waters has been rebooted and returned to service at 5:55PM following a power interuption earlier this afternoon. All running jobs were lost due to the outage.

    Please email help+bw@ncsa.illinois.edu to report any issues.

  • Blue Waters: NPCF Power Issue, Scheduler paused expect full reboot

    Created by: tbouvet 2019-07-05 13:32:15 (Channels: unscheduledoutages|systemnotices)

    A power outage at the building housing the Blue Waters system has caused a service interruption; the Login Nodes, Network, Storage, Compute Nodes, and Near-line Storage may be unavailable. It is unknown at this time when a return to service can be expected. Watch the Blue Waters portal blog for updates.

  • Nearline Tape Library Has Returned To Service

    Created by: briandi 2019-06-28 13:27:38 (Channels: unscheduledoutages|systemnotices)

    The NCSA_Nearline storage subsystem issue on Blue Waters was resolved and the system returned to normal operations at 1:00 pm.

     

  • Nearline Tape Library Emergency System Maintenance

    Created by: briandi 2019-06-28 09:45:21 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users, Blue Waters is experiencing an issue on the HPSS storage subsystem (ncsa#Nearline) that began at 9:00 AM. System support staff are evaluating and attempting to restore normal service. The rest of Blue Waters subsystems remain in normal operation. Transfers into or out of the system will be unavailable until...
    Read More
  • Blue Waters: NPCF Facility Power Maintenance Complete

    Created by: tbouvet 2019-06-24 06:07:40 (Channels: scheduledoutages|systemnotices)

    The NPCF facility power maintenance is complete and Blue Waters has returned to service at 06:00 hours.

    Please email help+bw@ncsa.illinois.edu to report any issues.

     

  • Blue Waters Scheduled Maintence Extended until June 24 6AM CT

    Created by: tbouvet 2019-06-24 05:18:12 (Channels: scheduledoutages|systemnotices)

    The scheduled maintenance has been extended until June 24th 6AM due to technical difficulties. We apologize for the delay in Return to Service. Availability details: Subsystem / Services  Projected Interruption Duration Compute June 23 05:00 - June 24 06:00 Login / Filesystem June 23 05:00 - June 24 06:00 Scheduler June 23 05:00 - June 24...
    Read More
  • Reminder: Blue Waters Scheduled Maintenance June 23rd 2019 5AM

    Created by: tbouvet 2019-06-22 14:39:23 (Channels: scheduledoutages|systemnotices)

    Reminder of Maintenance, On Sunday June 23, electrical maintenance is scheduled to take place at the NPCF Building which hosts Blue Waters.  Therefore, beginning at 05:00 CT, all Blue Waters subsystems will be unavailable.  The outage is expected to last until 03:00 the following morning, June 24. During this interruption, Blue Waters and...
    Read More
  • Blue Waters: Nearline Tape Library Has Returned To Service

    Created by: tbouvet 2019-05-31 12:58:05 (Channels: unscheduledoutages|systemnotices)

    The Nearline subsystem reboot completed at 11:30AM today.

    All existing transfers should resume as Nearline was returned to service.

  • Nearline Tape Library System Reboot

    Created by: bbode 2019-05-31 09:28:04 (Channels: unscheduledoutages|systemnotices)

    The Nearline subsystem is currently undergoing an emergancy reboot to clear multiple issues. It is expected to return to service by 1PM today.

    All existing transfers will resume once Nearline returns to service.

  • Blue Waters Has Returned to Service

    Created by: tbouvet 2019-05-16 17:15:55 (Channels: unscheduledoutages|systemnotices)

    The storage issue on Blue Waters projects file system has been resolved and the system returned to normal operations at "5:07" PM CT. Any teams who were impacted by the file system issue have been contacted individually. The scheduler has resumed normal operations. Thank you for your patience while this was...
    Read More
  • Blue Waters Project File System Update

    Created by: tbouvet 2019-05-16 09:29:40 (Channels: unscheduledoutages|systemnotices)

    We are in the process of running a file system check and repair of a small portion of the projects file system. When that is complete we will access the results and take appropriate action. Update: File system repair continues and is expected to last until late this afternoon (5/16). If...
    Read More
  • Blue Waters Project File System Issue

    Created by: tbouvet 2019-05-15 15:20:25 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users, Blue Waters is currently experiencing a storage server issue for a portion of the projects filesystem. As a result, all I/O transactions targeting the affected storage server will block. A single storage server supplies a small fraction of the file system data and all remaining storage servers continue normal operation....
    Read More
  • Blue Waters OSS Return to Service

    Created by: squaire3 2019-05-12 12:33:07 (Channels: unscheduledoutages|systemnotices)

    The storage issue on Blue Waters was resolved and the system returned to normal operations at 12:30 PM CT. Scheduler has also been resumed.

  • Blue Waters OSS Failover

    Created by: squaire3 2019-05-12 09:18:53 (Channels: unscheduledoutages|systemnotices)

    Blue Waters is currently experiencing a storage server failover on the OSS file system that began at 8:08 AM CT. As a result, all I/O transactions targeting the affected storage server will block until the failover completes on all clients. A single storage server supplies a small fraction of the...
    Read More
  • Nearline Tape Library System Maintenance

    Created by: glasgow 2019-04-29 16:55:33 (Channels: unscheduledoutages|systemnotices)

    Nearline maintenance work is complete and the service has returned to full operations as of: 1600hrs, April 29th, 2019 --- Nearline is undergoing emergency service on one tape library. The work is related to fallout from last week's hardware service and is expected to take approximately 10 hours to complete. Some files may...
    Read More
  • Nearline Tape Library System Maintenance

    Created by: glasgow 2019-04-29 09:42:49 (Channels: unscheduledoutages|systemnotices)

    Nearline is undergoing emergency service on one tape library. The work is related to fallout from last week's hardware service and is expected to take approximately 10 hours to complete. Some files may not be accessible during that time.

    Start time: 0945hrs to ~ 2000hrs, April 29, 2019

  • Nearline Tape Library System Maintenance

    Created by: glasgow 2019-04-22 15:04:14 (Channels: scheduledoutages|systemnotices)

    One of Nearline's four tape libraries will undergo maintenance to correct power control problems. This library system will be unavailable for four hours while the work is conducted. Files stored in the library will be unavailable for staging during that time. Retrieval jobs from Globus will wait for the files...
    Read More
  • Blue Waters Return to Service

    Created by: squaire3 2019-04-12 15:41:38 (Channels: unscheduledoutages|systemnotices)

    The high-speed network issue on Blue Waters was resolved and the system returned to normal operations at 3:34 PM CT. Scheduler has also been resumed.

  • Blue Waters Scheduler Paused - HSN Issue

    Created by: squaire3 2019-04-12 15:02:55 (Channels: unscheduledoutages|systemnotices)

    Blue Waters is experiencing an issue on the high-speed network that began at 2:10 PM CT. System support staff are evaluating and attempting to restore normal service. Job scheduling is paused until the issue is resolved. The file systems and data transfer services are operating normally. Logins have been occationally hanging...
    Read More
  • Nearline Endpoint Paused for Storage Maintenance

    Created by: glasgow 2019-04-08 23:14:20 (Channels: unscheduledoutages|systemnotices)

    The Nearline endpoint has now been returned to normal operations.  The Blue Water's Nearline endpoint will be paused beginning at 1700hrs CDT. New and current user actions/requests will be paused and will resume normal activity when the endpoint is released. No user action is necessary. This maintenance...
    Read More
  • Nearline Endpoint Paused for Storage Maintenance

    Created by: glasgow 2019-04-08 16:27:20 (Channels: unscheduledoutages|systemnotices)

    The Blue Water's Nearline endpoint will be paused beginning at 1700hrs CDT. New and current user actions/requests will be paused and will resume normal activity when the endpoint is released. No user action is necessary. This maintenance window will be used to conduct resource management operations that have been deferred...
    Read More
  • Blue Waters Returned to Service

    Created by: tbouvet 2019-04-07 15:14:36 (Channels: unscheduledoutages|systemnotices)

    Blue WAters Users,

    The storage server issue on the scratch file system is resolved. I/O transactions initiated during the outage should have resumed when the Lustre target returned. Blue Waters has resumed normal operations.

  • Blue Waters Scheduler is paused 9:30 AM

    Created by: tbouvet 2019-04-07 10:30:00 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users, Blue Waters is currently experiencing a storage server issue for a small portion of the scratch filesystem. As a result, all I/O transactions targeting the affected storage server will block. A single storage server supplies a small fraction of the file system data and all remaining storage servers continue normal...
    Read More
  • BlueWaters cabinet failure

    Created by: mshow 2019-03-23 16:44:36 (Channels: unscheduledoutages|systemnotices)

    The cabinet has been restored.

  • BlueWaters repair complete

    Created by: mshow 2019-03-23 05:13:22 (Channels: unscheduledoutages|systemnotices)

    The cabinet has been restored

  • BlueWaters cabinet failure

    Created by: mshow 2019-03-23 03:46:37 (Channels: unscheduledoutages|systemnotices)

    A cabinet has shutdown resulting in job loss and an incomplete network configuration. It is unknown at this time when a return to service can be expected for that cabinet. Watch the Blue Waters portal blog for updates.

  • Blue Waters returned to service

    Created by: mshow 2019-02-24 03:41:58 (Channels: mandatory|unscheduledoutages|systemnotices)

    The high-speed network issue on Blue Waters was resolved and the system returned to normal operations at 3:30 AM CT.

  • Blue Waters High Speed Network issue

    Created by: mshow 2019-02-23 23:43:19 (Channels: unscheduledoutages|systemnotices)

    Blue Waters is experiencing an issue on the high-speed network that began at 9:48 PM CT. System support staff are evaluating and attempting to restore normal service. Job scheduling is paused until the issue is resolved. The file systems and data transfer services are operating normally. Interim updates will be posted on...
    Read More
  • Blue Waters Notice: Unplanned Reboot

    Created by: tbouvet 2019-01-12 09:58:18 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users, We experienced an issue that has the high speed network in an unrecoverable state. We have to reboot the system to recover and all running jobs will be lost. The login nodes and endpoints (ncsa#Nearline ncsa#BlueWaters) will remain available during the reboot. The current estimate for return to...
    Read More
  • Blue Waters Notice: Nearline System (HPSS) return to service

    Created by: tbouvet 2019-01-10 15:24:20 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users:

    The ncsa#Nearline endpoint is available and has returned to service at 3PM CT. We apologize for any inconvenience.

    -Blue Waters

  • Blue Waters Notice: Nearline System (HPSS) remains unavailable.

    Created by: tbouvet 2019-01-10 09:38:30 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users:

    The ncsa#Nearline endpoint is paused pending repairs suffered to systems during the data center power failure on 1/09/19. We apologize for any inconvenience.

    Please check back for an update on the situtation.

    -Blue Waters

     

  • Blue Waters Notice: System has returned to service

    Created by: jenos 2019-01-10 09:37:52 (Channels: unscheduledoutages|systemnotices)

      Blue Waters Users: The power outage at the NPCF building has been resolved.  All Blue Waters services are now available with the exception of the Nearline storage system, which will take a bit longer to recover.  The job scheduler has been resumed and login nodes have access re-enabled.  There will be...
    Read More
  • Blue Waters Notice: Power disruption

    Created by: jenos 2019-01-09 16:21:59 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users: Update 4:20pm: All systems up except Nearline.  Access remains restricted while performance tests complete. A power outage at the building housing the Blue Waters system has caused a service interruption; all running jobs were consequently terminated.  All Blue Waters subsystems have been affected and are currently out of...
    Read More
  • Blue Waters Notice: System returned to service

    Created by: jenos 2018-12-27 23:43:11 (Channels: unscheduledoutages|systemnotices)

    Blue Waters Users:

    The power outage at the Blue Waters building has been resolved and the compute nodes have been restarted. The Blue Waters system has been returned to normal operations at 11:40 PM CT. We apologize for any inconvenience.

     

  • Blue Waters Announcement: Winter Holiday services

    Created by: gbauer 2018-12-21 09:21:34 (Channels: systemnotices|general)

    Blue Waters partners; NCSA staff will observe the University of Illinois Winter break schedule starting 5pm Friday December 21, 2018, and will resume normal business hours on Wednesday, January 2nd, 2019. On-call staff will monitor Blue Waters and the service request system during this period, and respond to issues in a...
    Read More
  • Blue Waters Announcement: Ongoing Blue Waters Job Discounts

    Created by: gbauer 2018-11-12 21:44:41 (Channels: systemnotices|general|policychange)

    This is a reminder that the first period of job discounts ends on Thursday, November 15th. The second period starts on the 16th but with different criteria. Get your jobs submitted soon. Stop by the NCSA booth and say hi if you are at SC18. > The Blue Waters project is happy to...
    Read More
  • Blue Waters Discount Update

    Created by: jenos 2018-11-02 17:22:33 (Channels: systemnotices|general|policychange)

    The previously announced discounts have not changed; however, they have not been activated yet to be reflected in current charges. This is expected to be activated next week, but will be retroactively applied to the original start date of Nov 1, 2018.

     

  • Blue Waters Announcement: Blue Friday Job Discounts start today!

    Created by: jenos 2018-11-01 12:34:44 (Channels: systemnotices|general|policychange)

    The Blue Waters project is happy to announce two job discount periods as our way of saying thank you for another successful year of computing and the your help creating our 2017-2018 annual report.   There are two discount periods: Nov 1 - Nov 15: 50% charge discount on all queues, all jobs completing during...
    Read More
  • Blue Waters Notice: Temporary changes to shell stack size limit

    Created by: gbauer 2018-10-05 14:40:33 (Channels: systemnotices)

    Blue Waters partners, We would like to inform you of a change to the Blue Waters user environment that will be made today, Friday October 5th, at 2:30 pm Central time. A subsequent notice will be sent when the change are reverted. The change is: A. The shell stack size soft limit will be set to 2097152 kbytes and the...
    Read More
  • Globus SSH-based CLI End of Life

    Created by: gbauer 2018-08-01 15:19:49 (Channels: systemnotices)

    As of August 1, 2018 the SSH-based command-line interface (CLI) to Globus is end-of-life.

    Please see view-source:https://bluewaters.ncsa.illinois.edu/data-transfer-doc (scroll to the CLI section) and/or the Globus CLI page https://docs.globus.org/cli/ for more information on the new Python based CLI.

    Send email to help+bw@ncsa.illinois.edu for assistance, questions, etc.

  • Blue Waters Scheduled Maintenance: Return to Service

    Created by: gbauer 2018-07-16 19:43:38 (Channels: scheduledoutages|systemnotices|softwareupdate)

    Blue Waters Users, Blue Waters has been returned to service after a security update and programming environment update. CUDA 9.1 is now available as the default cudatoolkit but with gcc/4.9.3 remaining as the default GNU compiler. The default programming environment otherwise did not change to allow for an extended transition to...
    Read More
  • Blue Waters Scheduled Maintenance Update

    Created by: jenos 2018-07-13 11:57:54 (Channels: scheduledoutages|systemnotices|softwareupdate)

    Blue Waters Users, Please note the following scheduled maintenance update: The scheduled maintenance for CUDA 9.1 deployment on Blue Waters has been expanded to a near full system service outage that will span 13 hours on Monday July 16th, from 9am to 10pm. This adjustment will eliminate the need for an additional...
    Read More
  • Blue Waters XK (GPU) resource and HPSS Nearline resources will be unavailable from 9am to 12pm on Monday July 16th

    Created by: kingda 2018-07-09 09:08:10 (Channels: scheduledoutages|systemnotices|softwareupdate)

    Blue Waters Users, Blue Waters XK compute resource and HPSS ncsa#Nearline resource will be undergoing maintenance Monday (July 16th) morning beginning at 9 AM. The XK compute nodes are anticipated to be unavailable for up to 3 hours. The login nodes and XE compute resource will remain in service. A new Programming Environment will be installed...
    Read More
  • Blue Waters Partial Scratch Outage

    Created by: tbouvet 2018-06-23 21:31:02 (Channels: unscheduledoutages|systemnotices)

    Blue Waters experienced a newtork switch failure that resulted in a partial outage of the scratch filesystem (ost168-179) from 7:44 PM CT to 7:59 PM CT. Jobs that ended during this time may have been impacted. I/O transactions targeting the affected storage server should block until the ost targets returned...
    Read More
  • Blue Waters Nearline Endpoint Busy

    Created by: gbauer 2018-06-21 23:07:24 (Channels: systemnotices)

    Due to very high demand for data retrieval from Nearline, a pause rule is in effect to allow manual task scheduling. You may submit tasks as normal and they will be run as quickly as possible. Tasks submitted to Globus will start in a paused state but will be released to...
    Read More
  • Blue Waters returned to full service

    Created by: mshow 2018-06-12 10:23:37 (Channels: mandatory|unscheduledoutages|systemnotices)

    Blue Waters has returned to full service after recovery from a power event.

  • Blue Waters system power interruption

    Created by: mshow 2018-06-12 04:56:14 (Channels: mandatory|unscheduledoutages|systemnotices)

    Thunderstorms have resulted in a power interruption of the BlueWaters System. This outage imacts both the compute nodes and all  filesystems. Therefore, a full reboot will be necessary.Return to service is estimated to be approximately 10 am Centeral time.

    BW Admin

  • Blue Waters has Returned to Service

    Created by: tbouvet 2018-06-07 14:27:12 (Channels: unscheduledoutages|systemnotices)

    Blue Waters has returned to full service at 2:14 PM CT. The issue encountered required a full system reboot to resolve. All running jobs were lost so please resubmit your jobs from latest checkpoint file if your job exited prematurely.   

  • Blue Waters notice: Container software re-enabled on Blue Waters

    Created by: gbauer 2018-05-09 16:25:31 (Channels: systemnotices|softwareupdate)

    We are pleased to announce that the container software package Shifter and its associated service have been re-enabled on Blue Waters. Shifter was checked for vulnerabilities and none were found. To avoid concern, a patch was applied, and Shifter has been updated and re-enabled. Please see the Shifter page on the portal for...
    Read More
  • Blue Waters notice: Container software disabled on Blue Waters

    Created by: gbauer 2018-05-02 14:28:53 (Channels: systemnotices)

    This notice affects Blue Waters users relying on the container technologies of Shifter or Singularity.  Due to a recently announced security vulnerability related to container software, Shifter and Singularity container software have been disabled immediately to address the issue. We took this action to eliminate the risk of potential compromise of Blue Waters...
    Read More
  • Blue Waters Announcement: Winter Holiday services

    Created by: gbauer 2017-12-21 15:49:34 (Channels: systemnotices|general)

    Blue Waters partners; NCSA staff will observe the University of Illinois Winter break schedule starting 5pm Friday December 22, 2017, and will resume normal business hours on Tuesday, January 2nd, 2018. On-call staff will monitor Blue Waters and the service request system during this period and respond to issues in a...
    Read More