Sciense Oprations

The Gratia Database is currently experiencing an outage. On Friday, April 25, the GOC reported instability in some of the services that rely on Gratia for information. Over the course of the weekend, this instability degraded to outage status. Fermilab is currently working to tune the Gratia database. Maintenance is ongoing and there is no estimated time of repair.



We will pass along updates as they become available to us.



During this time Gratia will continue to collect and store accounting records normally and local Gratia record caching will occur if the collector reaches it limits, however no updates will occur to the above referenced sites until it is fully restored.

GratiaWeb is currently experiencing an instability. Additionally, OSG Display is experiencing delays with updates. We are currently investigating the cause of the instability. More information can be found here https://ticket.grid.iu.edu/20881. We will provide updates as they become available.



Please contact the GOC with any questions.

Gratia DB resumed normal operations at 09:15 GMT this morning. Gratiaweb and OSG Display have also resumed normal operations and Gratia-APEL will follow shortly.



Please contact the GOC with any questions.

Fermilab is still working on the gratia DB failures reported earlier. Maintenance is ongoing at this point and no estimated time of repair is available. We will pass on updates as they are avaliable. As a reminder, gratiaweb.opensciencegrid.org and display.opensciencegrid.org will be affected. During this time Gratia will continue to collect and store accounting records normally and local Gratia record caching will occur if the collector reaches it limits, however no updates will occur to the above referenced sites until it is fully restored.

The GOC announces a special maintenance period starting Tuesday, April 29th, 2014 at 13:00 UTC. The GOC reserves 8 hours in the unlikely event that unexpected problems are encountered.



Oasis

Upgrade oasis-replica to cvmfs-server v2.1. Only the GOC instance will be affected.

The Gratia accounting database investigation has completed and a restoration plan is underway. Due to the size of the database we expect it to remain unavailable for publishing data to gratiaweb.opensciencegrid.org and display.opensciencegrid.org until at least tomorrow afternoon. During this time Gratia will continue to collect and store accounting records normally and local Gratia record caching will occur if the collector reaches it limits, however no updates will occur to the above referenced sites until it is fully restored. We will inform the community when Gratia is back to normal operation.

Due to a hardware failure the Gratia accounting database has become corrupted. This is currently affecting the publishing of new information to gratiaweb.opensciencegrid.org and display.opensciencegrid.org. The Gratia collector is still functioning normally and all records are being collected and preserved. We are investigating the best way to restore the data from backup and process the new records held on the collector. Until we have completed this emergency maintenance no new data will be published to the sites referenced above.

The GOC will upgrade the following services beginning Tuesday, April 22nd, 2014 at 13:00 UTC. The GOC reserves 8 hours in the unlikely event that unexpected problems are encountered. We encourage users to test affected services before the production release.



GratiaWeb v1.2-28



Rebuilding with RHEL6 image (new version may require python2.6)

Moved all external ‘URL’s into a single file location for easier update and tracking.

Reworked WLCG Reporting (Overview) to contain all summary pledge and capacity information by Federation Name and Resource Group

Reworked WLCG Reporting (Detailed) to contain detailed resources supplied information by Federation Name and Resource Group

New WLCG report pages

Bug fix to ATLAS data transfer graphing

Improvement to wlcg reports error handling

Completion of move for all GratiaWeb URLs to a config file

graphtool / Reworked RPM packaging

graphtool / Updated database connection error handling



GOC-TX 1.36



Added fp114_fermide TX ID for test purpose.



GOC Ticket 1.74

(Didn’t get released during the last release)



Made Gratia software ticket to go to GOC support staff instead of SNOW-GratiaDev (https://ticket.grid.iu.edu/20235)

Oasis



Upgrade oasis-replica to cvmfs-server v2.1. Only the GOC instance will be affected.



All Services



There will be OS updates; reboots will be required. Downtime should be minimal, and the usual high-availability mechanisms will be used to reduce service downtime even further and eliminate it in most cases. However, services may experience degraded performance, and the services without HA mechanisms (OIM and Twiki) will still experience brief downtimes.

Greetings,



On Thursday, April 17th, the gratia-reporters service will be

decommissioned. We recommend anyone still using this service to switch

to the GratiaWeb interface at http://gratiaweb.opensciencegrid.org/. You

should be able to search all gratia data there, however if you need to

do a local search there are local gratia collector instances available at

http://gratia-fermi-osg.fnal.gov:8100/gratia

http://gratia-fermi-transfer.fnal.gov:8100/gratia

http://gratia-fermi-itb.gov:8100/gratia



thanks,

-Kevin

Tomorrow, April 9th, starting at 9:00 (eastern) the GOC will perform emergency maintenance on some services. The usual

high-availability mechanisms will be used to eliminate visible downtime for all services except Oasis where a brief

outage should be expected.

We are pleased to announce OSG Software versions 3.1.32 and 3.2.8.



These releases contain updated Certificate Authority (CA) bundles.

It is imperative that these new CA packages are installed before

June 1st. At that time, DigiCert will begin issuing OSG certificates

that depend on the new CA bundles.



OSG 3.1.32 and 3.2.8 contain:

* Updated CA certificates (IGTF 1.56)

- New SHA-2 signed DigiCert CA certificates

- DOEGrids CA certificates removed

- Old format CA certificate bundles discontinued

* Update to CVMFS 2.1.17 (many enhancements and bug fixes)

* VO Package v52

- New Sub VOs: Lariat, Gendetrd, Lar1, and Okra

- New VOMS servers at CERN

* Bug fixes for VOMS admin

* Updated gratia probes (sge, lsf, htcondor)

* Update to RSV 3.7.15 (bug fixes)

* Update to MyProxy 5.9 (bug fixes)

* Many other minor bug fixes



OSG 3.2.8 also contains:

* Bug fix for crash in gridftp-hdfs



Release notes and pointers to more documentation can be found at:



https://www.opensciencegrid.org/bin/view/Documentation/Release3/Release328

https://www.opensciencegrid.org/bin/view/Documentation/Release3/Release3132



Need help? Let us know:



https://www.opensciencegrid.org/bin/view/Documentation/Release3/HelpProcedure



We welcome feedback on this release!

Applications for the 2014 OSG User School are due soon. To make it easier

to apply, we extended the application deadline as late as possible: Sunday,

April 13th. But there is no reason to wait ... apply today for acceptance

to the School and for financial support!





ANNOUNCING THE 2014 OPEN SCIENCE GRID USER SCHOOL!



If you could access thousands, maybe millions of hours of computing, how

would it transform your research? What discoveries would you make?



We are looking for qualified students to attend the 2014 Open Science Grid

(OSG) User School, where they will learn how to use high throughput

computing (HTC) to harness vast amounts of computing power for research.



Using lectures, discussions, roleplays, and lots of hands-on work with OSG

experts in high throughput computing, students will learn how HTC systems

work, how to run and manage many jobs and huge datasets to implement a full

scientific computing workflow, and where to turn for help and more info.



Worried about costs? Successful applicants will get financial support to

attend the OSG School (July 7-10) at the beautiful University of Wisconsin

in Madison.



Ideal candidates are science, technology, engineering, and mathematics

(STEM) graduate students whose research demands large-scale computing.

Also, we will consider applications from faculty, staff, and advanced

undergraduates, so make a good case for yourself!



IMPORTANT DATES



Application Period: March 10 - April 13 (EXTENDED!)

OSG User School: July 7-10



MORE INFORMATION AND APPLICATIONS



Web: http://www.opensciencegrid.org/UserSchool

Email: osg-school-2014@opensciencegrid.org

Facebook: https://www.facebook.com/OSGUserSchool

Twitter: https://twitter.com/OSGUserSchool



Please forward this announcement to help us reach potential students. And

consider posting our flyer where appropriate:



https://twiki.opensciencegrid.org/twiki/pub/Education/OSGUserSchool2014/2014-osg-user-school-flyer.pdf