Last Update 2019-07-18 11:47 UTC , execution time: 381s,      

Slots Running (10-11 UTC)379k  = 92k SCORE  + 287k MCORE 

8h avg:366k;24h avg:393k

Removed:src_endpoint=*LAKE*;dst_endpoint=*LAKE*

Transfer Failure rate (last hour):  0.94 f/s 8h avg: 1.90 f/s;24h avg: 2.17 f/s

Transfer Throughtput (last hour):  14.15 GB/s 8h avg: 17.52 GB/s;24h avg: 19.28 GB/s

Transfer Rate (last hour)14.15 f/s 8h avg: 15.42 f/s;24h avg: 18.27 f/s

Transfer Efficiency (<60%, 4h):  IT (1h)  

Services in Warning (4h):

Services in Alarm (4h): Panda Servers Rucio Deamons Automatix Rucio Deamons Necromancer Rucio Deamons Transmogrifier

Storage: TAPE (0) T0 (0) T1 (0) T2 (1) T0s (0) T1s (1) T2s (0)

T0 slots: 10899

T0 batch usage 

Doc

Production queues

Full queues [39]

 

Empty queues [331]

 

Excessive assigned[38]

 

Zero running [18]

 

Analysis queues

Excessive activated[1]

 

Zero running [0]

 

Site Details

Degradation summary:

  • cvmfs_stratum1_atlas
    • timestamp:2019-07-17T15:15:00Z; availability:50; availabilityinfo:
      1. Stratum 1 on cvmfs.fnal.gov is 10 releases behind and has not been updated 373 seconds.
    • timestamp:2019-07-17T15:05:00Z; availability:50; availabilityinfo:
      1. Stratum 1 on cernvmfs.gridpp.rl.ac.uk is slightly behind the stratum zero by 341 seconds and 1 release(s).
      2. Stratum 1 on cvmfs.fnal.gov is 8 releases behind and has not been updated 342 seconds.
    • timestamp:2019-07-17T14:55:00Z; availability:0; availabilityinfo:
      1. Stratum 1 on cernvmfs.gridpp.rl.ac.uk is 6 releases behind and has not been updated 199 seconds.
      2. Stratum 1 on cvmfs.fnal.gov is 7 releases behind and has not been updated 199 seconds.
      3. Stratum 1 on cvmfs.racf.bnl.gov is slightly behind the stratum zero by 199 seconds and 2 release(s).
    • timestamp:2019-07-17T14:45:00Z; availability:50; availabilityinfo:
      1. Stratum 1 on cernvmfs.gridpp.rl.ac.uk is slightly behind the stratum zero by 923 seconds and 4 release(s).
      2. Stratum 1 on cvmfs.fnal.gov is 5 releases behind and has not been updated 923 seconds.
    • timestamp:2019-07-17T14:35:00Z; availability:50; availabilityinfo:
      1. Stratum 1 on cernvmfs.gridpp.rl.ac.uk is slightly behind the stratum zero by 321 seconds and 4 release(s).
      2. Stratum 1 on cvmfs-stratum-one.cern.ch is slightly behind the stratum zero by 321 seconds and 1 release(s).
      3. Stratum 1 on cvmfs.fnal.gov is 5 releases behind and has not been updated 322 seconds.
  • pandacache_aipanda012
    • timestamp:2019-07-18T02:00:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
  • pandacache_aipanda047
    • timestamp:2019-07-18T10:55:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:50:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:45:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:40:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:35:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:30:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:25:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:20:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:15:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:10:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:05:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:00:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:55:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:50:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:45:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:40:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:35:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:30:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:25:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:20:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:15:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:00:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T08:55:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T08:50:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T08:45:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T08:40:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T08:35:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T08:30:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T01:20:00Z; availability:0; availabilityinfo:PanDA cache unresponsive: no answer to isAlive request
  • pandajedi_aipanda041
    • timestamp:2019-07-18T10:48:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:33:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:18:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:03:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T09:48:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T09:03:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T08:48:00Z; availability:0; availabilityinfo:JEDI process not found
  • pandajedi_aipanda042
    • timestamp:2019-07-18T10:48:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:33:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:18:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:03:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T09:48:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T09:33:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T08:48:00Z; availability:0; availabilityinfo:JEDI process not found
  • pandajedi_aipanda186
    • timestamp:2019-07-18T10:48:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:33:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:18:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T10:03:00Z; availability:50; availabilityinfo:JEDI is OK, but filebeat service is not running
    • timestamp:2019-07-18T09:48:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T09:33:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T09:03:00Z; availability:0; availabilityinfo:JEDI process not found
    • timestamp:2019-07-18T08:48:00Z; availability:0; availabilityinfo:JEDI process not found
  • pandaserver_aipanda052
    • timestamp:2019-07-18T10:55:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:50:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:45:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:40:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:35:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:30:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:25:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:20:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:15:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:10:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:05:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:00:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:55:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:50:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:45:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:40:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
  • pandaserver_aipanda055
    • timestamp:2019-07-17T17:18:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
  • pandaserver_aipanda189
    • timestamp:2019-07-18T10:55:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:50:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:45:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:40:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:35:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:30:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:25:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:20:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:15:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:10:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:05:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T10:00:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:55:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:50:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:45:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:40:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
    • timestamp:2019-07-18T09:30:00Z; availability:0; availabilityinfo:PanDA server unresponsive: no answer to isAlive request
  • rucio.daemons.automatix
    • timestamp:2019-07-18T08:40:00Z; availability:0; availabilityinfo:UNKNOWN
  • rucio.daemons.necromancer
    • timestamp:2019-07-18T08:41:00Z; availability:0; availabilityinfo:UNKNOWN
  • rucio.daemons.transmogrifier
    • timestamp:2019-07-18T08:41:00Z; availability:0; availabilityinfo:UNKNOWN

Availability overview:

12 13 14 15 16 17 18 19 20 21 22 23 00 01 02 03 04 05 06 07 08 09 10 11
Hours (UTC)
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
AGIS API
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
AGIS Web UI
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
AMI
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
ARC Control Tower
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
EventIndex Consumer
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
EventIndex Hadoop Cluster
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
EventIndex Web
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Frontier
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Global CVMFS ATLAS CONDB
100 100 100 0 50 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Global CVMFS Stratum1 ATLAS
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Panda Harvester
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Panda Logstash
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Panda Proxy
100 100 100 100 100 100 0 100 100 100 100 100 100 100 0 0 100 100 100 100 100 0 0 0
Panda Servers
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Pilot Factory
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 0 100 100
Rucio Deamons Automatix
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Rucio Deamons Conveyor
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Rucio Deamons Hermes
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Rucio Deamons Judge
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Rucio Deamons Kronos
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 0 100 100
Rucio Deamons Necromancer
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Rucio Deamons Reaper
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 0 100 100
Rucio Deamons Transmogrifier
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Rucio Deamons Undertaker
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Rucio HTTPD
posted by Tomas Blazek on Thu, Jul 18, 2019, 01:44 +0200
posted by Claire Adam-Bourdarios on Wed, Jul 17, 2019, 15:53 +0200
posted by Tomas Blazek on Wed, Jul 17, 2019, 00:08 +0200
posted by Claire Adam-Bourdarios on Tue, Jul 16, 2019, 15:49 +0200
posted by Smita Darmora on Tue, Jul 16, 2019, 00:00 +0200
posted by Claire Adam-Bourdarios on Mon, Jul 15, 2019, 16:00 +0200