Follow up of this weekend's cooling water incident

Dear labmembers --

Thanks for your patience this weekend as the campus struggled with loss
of cooling water. Several people have asked what happened. This is what
we understand.

Just before 3 pm Friday, a 60Kv electrical line was damaged during
construction work, causing the Cardinal Cogeneration Plant to shut down.
Those of you in the lab at that time may have experienced the associated
power glitch which caused a number of tools to shutdown or lockup. The
Cogen plant is the large green "steamy" structure across the street from
us. It supplies power, cooling water and steam to the entire Stanford
campus. Cogen cooling water is used to absorb heat from the Allen
building process cooling water (PCW) recirculating system. SNF uses PCW
to cool pumps, chillers, and other heat-generating systems. Cogen
cooling water is also used for controlling building temperature and fab
humidity and temperature. Without cooling water, our tools shut off and
the building/lab heat up. Friday afternoon, campus wide, buildings got
hot and computer system shut down.

Emergency backup systems kicked in. In the case of SNF, city water was
automatically introduced to keep the PCW temperatures down. However,
with the whole campus also using city water as backup, the pressure was
not sufficient to keep our tools running. By Friday evening, SNF staff
were able to safety shutdown the major heat generating tools and thus
avoid damage to tools or the PCW system. Stanford emergency crews worked
through the weekend and by around 9 pm Sunday evening were able to
restore cooling water campus wide. Although a couple of SNF tools are
still suffering from the unexpected shutdown, quick action and long
hours by SNF and FacOps staff prevented serious damage. Special thanks
to Maurice, Mahnaz, Ed, Ray, and Elmer for directing traffic and
shutting tools down; Jim H for cutting his vacation short to come in,
help and advise; Ted for calling in instructions and advice while home
sick; and Tony in FacOps for keeping staff informed and putting in long
hours to make sure the PCW was OK.

While recognizing the heroic efforts of many staff, there are a number
of things we need to do to improve communications at the infrastructure
level and with our labmember community. These will be addressed in
coming weeks. If there are any questions about specific tools, please
consult with Coral and the responsible staff member. If there are
questions about this incident, please don't hesitate to ask.

Thanks again,

Your SNF Staff