CERN VOMS service status Fall-to-End 2005

and in the last 6 months as requested by Maite Barroso in the Dec. 9th 2005 meeting agenda (attached). Actions and conclusions from that meeting can be found here. VOMS testing status reports are indexed in https://edms.cern.ch/document/631389/

Tomcat performance: This was *the* show-stopper between June 8th and December 7th when the tomcat *symptom* was diagnosed by the developer to be due to a bug in voms-admin. So far, users were unable to register, VO managers unable to work with the server and sites unable to make the grid-map file. Many nested savannah tickets, latest ticket_id=14057. The problem has been repetitively reported since last spring, for examples please see point 4 in the Executive summary of the May 23rd meeting and escalation to the LCG/gLite management. Hourly restarts of tomcat and tripling the physical memory of the primary server didn't solve the problem.
voms bugs emerging in new code that fixes other bugs: Please look at the bug list in the latest gLite Release notes, section 5. Given that gLite R1.2, R1.3 and R1.4 were issued during the last 6 months, one can see the number of bugs concerned. Examples that particularly affected the LCG VOMS production service:

a long list of savannah tickets in point 3 of the Executive summary of the Aug.9th VOMS migration checkpoint meeting.
ticket id=13899 (voms core bug: affects user clients and command line tools)
ticket id=14114 (voms core bug: details in D.Smith's attached message)
ticket id=10970 (voms-admin bug: didn't close database connections. Gridmap file generation hang and site admins complained. First observed in gLite R1.3. Fixed in R 1.4)
ticket id=13863 (voms-admin bug: sometimes gets stuck in the boot process and locks down tomcat. All VOs found locked on Mon Nov 21st. First observed in gLite R1.4)
ticket id=13592 and ticket id=10813 (voms-admin bugs: concern pending requests for processing by the VO-Admin. Don't affect VOMRS-based Registration flow. First observed in gLite R1.4)

VOMS data migration to the Production Oracle database servers can't be satisfied due to the lack of connection pooling in voms core. Details in ticket ticket id=12613. This is also a show-stopper due to the criticality of service, therefore the requirements for data integrity and availability.
Proper system administration provision (by CERN-IT-FIO). This was officially requested in November 2004 (see supporting document) but was a 'victim' of multiple IT re-organisations so far and starts now making progress.

The VOMS service requirements are listed in a detailed questionnaire linked from the LCG Fabric Tasks Dashboard

Other recent bugs/improvements:

ticket id=12590 (improvement: facilitate tomcat parameters for voms server managers)
ticket id=14026 (enhancement: require the voms server host certificate DN only on voms clients)
ticket id=14106 (enhancement: allow small vomslife values)

Hostnames and VOMS versions installed at CERN as of October 21st 2005

lcg-voms.cern.ch

voms.cern.ch

voms-slave.cern.ch

voms-test.cern.ch

Runs
glite voms-oracle R1.4.1 and VOMRS 1.2
This is the primary server. VODB is built using VOMRS and ORGDB link.

VOs defined:
alice
atlas
cms
lhcb
dteam
sixt
unosat
geant4
test

Data reside on grid8.cern.ch. There are 100 entries across VOs is this db now.

Physical memory: 1.5GB

Runs glite voms-oracle R1.4.1.
VODB populated via ldap-sync.

VOs defined:
alice
atlas
cms
lhcb
dteam

Data reside on grid8.cern.ch (different db) . There are 1500 entries across VOs is this db now.

Physical memory: 0.5GB

This is the hot-spare for lcg-voms.cern.ch.

Runs
glite voms-oracle R1.4.1, and VOMRS 1.2.

Identical configuration and VOs as on lcg-voms.cern.ch.

The Oracle db is shared with lcg-voms.cern.ch.

Physicalmemory:0.5GB

Runs
glite voms-oracle R1.4.1.
This is a TEST machine not visible from outside CERN.

It can crash without impact on the service.

Physicalmemory:0.125GB

Maria Dimou, IT/GD, Grid Infrastructure Services