Diff of Observing Log/2014-07-31/message_restart_cosmos_all

• The added line is THIS COLOR.
• The deleted line is THIS COLOR.
• Observing Log/2014-07-31/message_restart_cosmos_all has been deleted.
• Deleting diff of Observing Log/2014-07-31/message_restart_cosmos_all

[[Observing Log/2014-07-31]]
#author("2020-04-02T09:30:45+09:00","","")

-2014-07-31 UT 02:45- UT03:00(cntl-C)
 restart_cosmos_all (at aste-mx1)
-Message on terminal;
#pre{{
 ***** aste-2c *****
/opt/OpenSSH/bin/ssh -f aste-2c /cosmos3/aste/nro/rc3.d/S99astnrod stop
/opt/OpenSSH/bin/ssh -f aste-2c /cosmos3/aste/rc3.d/S99astd stop
sleep 20
/cosmos3/aste/nro/rc3.d/S99astscam_shm
Stopping scam_shm_server 18083
/cosmos3/aste/nro/rc3.d/S99antlog
Stopping antlog 18094 18306
/cosmos3/aste/nro/rc3.d/S99astscam_mon
Stopping scam_stat_mon.exe 18107 18106
/cosmos3/aste/nro/rc3.d/S99astscam_vkput
Stopping scam_shm_vkput.exe 18120 18308
FACILITY=0 /cosmos3/aste/rc3.d/S99astpcv
Stopping astpcv
FACILITY=0 /cosmos3/aste/rc3.d/S99astcstgb
Stopping cstgb
FACILITY=1 /cosmos3/aste/rc3.d/S99astactd
Stopping the Local watcher.
FACILITY=1 /cosmos3/aste/rc3.d/S99astqllsv
Stopping the Quick Look Server.
FACILITY=1 /cosmos3/aste/rc3.d/S99astqlcsv
Stopping the Quick Look Server.
FACILITY=2 /cosmos3/aste/rc3.d/S99astkyr
Stopping KeyWord Resolver daemon.
FACILITY=2 /cosmos3/aste/rc3.d/S99astmrgld
Stopping the local controler.
FACILITY=2 /cosmos3/aste/rc3.d/S99astmrg
Stopping the local controler.
FACILITY=3 /cosmos3/aste/rc3.d/S99astmng
18007
Stopping the Manager.
FACILITY=4 /cosmos3/aste/rc3.d/S99astchk
Stopping Communication watch daemon.
FACILITY=5 /cosmos3/aste/rc3.d/S99astlsv
Stopping the Auto Observation Server.
FACILITY=6 /cosmos3/aste/rc3.d/S99astcnt1
Stopping CONT_BE local controler.
FACILITY=6 /cosmos3/aste/rc3.d/S99astbol
Stopping BOL daemon.
FACILITY=6 /cosmos3/aste/rc3.d/S99astcnt2
Stopping CONT_HLG local controler.
FACILITY=6 /cosmos3/aste/rc3.d/S99asthlg
Stopping HOLOG daemon.
FACILITY=6 /cosmos3/aste/rc3.d/S99astcnt3
Stopping CONT2 local controler.
FACILITY=3 /cosmos3/aste/rc3.d/S99astmguid
Stopping the Manager GUI data send server.
***** aste-1s ***** [#f7dd3e07]
/opt/OpenSSH/bin/ssh -f aste-1s /cosmos3/aste/nro/rc3.d/S99astnrod stop
/opt/OpenSSH/bin/ssh -f aste-1s /cosmos3/aste/rc3.d/S99astd stop
sleep 20
FACILITY=2 /cosmos3/aste/rc3.d/S99astmrgd
Stopping the local controler.
FACILITY=7 /cosmos3/aste/rc3.d/S99astqlsv
Stopping QLSV local controller.
***** aste-3c ***** [#t979ad20]
/opt/OpenSSH/bin/ssh -f aste-3c /cosmos3/aste/nro/rc3.d/S99astnrod stop
/opt/OpenSSH/bin/ssh -f aste-3c /cosmos3/aste/rc3.d/S99astd stop
sleep 20
/cosmos3/aste/nro/rc3.d/S99mmc1
Stopping mmc1_server 18169 18197
/cosmos3/aste/nro/rc3.d/S99sisrxmon
FACILITY=1 /cosmos3/aste/rc3.d/S99astref1
Stopping SYNTHE_A local controler.
FACILITY=2 /cosmos3/aste/rc3.d/S99astref2
Stopping SYNTHE_B local controler.
FACILITY=3 /cosmos3/aste/rc3.d/S99astrxt
Stopping RXT local controler.
FACILITY=5 /cosmos3/aste/rc3.d/S99astmmc
Stopping MMC local controler.
FACILITY=3 /cosmos3/aste/rc3.d/S99astnrorxtd
Stopping RXTD daemon.
***** aste-v1c ***** [#z9446774]
/opt/OpenSSH/bin/ssh -f aste-v1c /cosmos3/aste/nro/rc3.d/S99astnrod stop
/opt/OpenSSH/bin/ssh -f aste-v1c /cosmos3/aste/rc3.d/S99astd stop
sleep 20
/cosmos3/aste/nro/rc3.d/S98shmserv
/cosmos3/aste/nro/rc3.d/S99astwebcamd
webcam daemon stop.
/cosmos3/aste/nro/rc3.d/S99shmgen
Stopping shm_gen 16201 16200
/cosmos3/aste/nro/rc3.d/S99shmcabin
Stopping shm_cabin 16212 16147 16213
/cosmos3/aste/nro/rc3.d/S99shmshelter
Stopping shm_shelter 16165 16164
/cosmos3/aste/nro/rc3.d/S99shmshelter: kill: no such process
/cosmos3/aste/nro/rc3.d/S99shmconfroom
Stopping shm_confroom 16176 16177
/cosmos3/aste/nro/rc3.d/S99weather
/cosmos3/aste/nro/rc3.d/S99shmgenlog
/cosmos3/aste/nro/rc3.d/S99shmcabinlog
/cosmos3/aste/nro/rc3.d/S99shmenvlog
Stopping shm_envlog 16278 16281
/cosmos3/aste/nro/rc3.d/S99shmweatherlog
Stopping shm_weatherlog 16291 16290
/cosmos3/aste/nro/rc3.d/S99shmmisclog
Stopping shm_misclog 16302 16305
/cosmos3/aste/nro/rc3.d/S99shmtipper
Stopping shm_taumon 16315 16314
/cosmos3/aste/nro/rc3.d/S99shmtipper: kill: no such process
/cosmos3/aste/nro/rc3.d/S99shmvkput
/cosmos3/aste/nro/rc3.d/S99antcadac22
/cosmos3/aste/nro/rc3.d/S99antcadac22_log
/cosmos3/aste/nro/rc3.d/S99antcadac21_log
***** aste-c1c ***** [#y9b69250]
/opt/OpenSSH/bin/ssh -f aste-c1c /cosmos3/aste/nro/rc3.d/S99astnrod stop
/opt/OpenSSH/bin/ssh -f aste-c1c /cosmos3/aste/rc3.d/S99astd stop
sleep 20
/cosmos3/aste/nro/rc3.d/S99antcadac
Stopping cadac6_loop 435 455
/cosmos3/aste/nro/rc3.d/S99antcadac21
***** aste-1c ***** [#obbb1ea8]
/opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/nro/rc3.d/S99astnrod stop
/opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/rc3.d/S99astd stop
sleep 20
/cosmos3/aste/nro/rc3.d/S98asp_shm
Stopping asp_shm_server 9142 9139
/cosmos3/aste/nro/rc3.d/S99antstat
/cosmos3/aste/nro/rc3.d/S99aspirin
/cosmos3/aste/nro/rc3.d/S99asp_log
/cosmos3/aste/nro/rc3.d/S99asp_watchdog
/cosmos3/aste/nro/rc3.d/S99antcadac2asp
/cosmos3/aste/nro/rc3.d/S99autoweather
Stopping autoweather 9254 9216
/cosmos3/aste/nro/rc3.d/S99wind2shmasp
/cosmos3/aste/nro/rc3.d/S99wind2asp
FACILITY=0 /cosmos3/aste/rc3.d/S99astant
Stopping ANTENNA_G local controler.
FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvm
Stopping astpcvm
FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvs
Stopping astpcvs
***** aste-2c ***** [#ta79d013]
/opt/OpenSSH/bin/ssh -f aste-2c /cosmos3/aste/rc3.d/S99astd start
/opt/OpenSSH/bin/ssh -f aste-2c /cosmos3/aste/nro/rc3.d/S99astnrod start
sleep 20
FACILITY=0 /cosmos3/aste/rc3.d/S99astpcv
astpcv starting.
FACILITY=0 /cosmos3/aste/rc3.d/S99astcstgb
cstgb starting.
FACILITY=1 /cosmos3/aste/rc3.d/S99astactd
Local watcher starting.
FACILITY=1 /cosmos3/aste/rc3.d/S99astqllsv
Quick Look Server starting.
FACILITY=1 /cosmos3/aste/rc3.d/S99astqlcsv
Quick Look Server (continuum) starting.
FACILITY=2 /cosmos3/aste/rc3.d/S99astkyr
KeyWord Resolver daemon starting.
FACILITY=2 /cosmos3/aste/rc3.d/S99astmrgld
Local controler starting.
FACILITY=2 /cosmos3/aste/rc3.d/S99astmrg
Local controler starting.
FACILITY=3 /cosmos3/aste/rc3.d/S99astmng
Manager starting.
FACILITY=4 /cosmos3/aste/rc3.d/S99astchk
Starting communication watch daemon.
FACILITY=5 /cosmos3/aste/rc3.d/S99astlsv
Auto Observation Server starting.
FACILITY=6 /cosmos3/aste/rc3.d/S99astcnt1
Starting CONT_BE local controler.
FACILITY=6 /cosmos3/aste/rc3.d/S99astbol
Starting BOL daemon.
FACILITY=6 /cosmos3/aste/rc3.d/S99astcnt2
Starting CONT_HLG local controler.
FACILITY=6 /cosmos3/aste/rc3.d/S99asthlg
Starting HOLOG daemon.
FACILITY=6 /cosmos3/aste/rc3.d/S99astcnt3
/cosmos3/aste/nro/rc3.d/S99astscam_shm
Starting CONT2 local controler.
FACILITY=3 /cosmos3/aste/rc3.d/S99astmguid
Starting scam_shm_server
/cosmos3/aste/nro/rc3.d/S99antlog
Manager gui data send server starting.
Starting: antlog
/cosmos3/aste/nro/rc3.d/S99astscam_mon
Starting scam_stat_mon.exe
/cosmos3/aste/nro/rc3.d/S99astscam_vkput
Starting scam_shm_vkput.exe
***** aste-1s ***** [#ud5cfe3b]
/opt/OpenSSH/bin/ssh -f aste-1s /cosmos3/aste/rc3.d/S99astd start
/opt/OpenSSH/bin/ssh -f aste-1s /cosmos3/aste/nro/rc3.d/S99astnrod start
sleep 20
FACILITY=2 /cosmos3/aste/rc3.d/S99astmrgd
Local controler starting.
FACILITY=7 /cosmos3/aste/rc3.d/S99astqlsv
Starting QLSV local controller.
***** aste-3c ***** [#xb8ef8a4]
/opt/OpenSSH/bin/ssh -f aste-3c /cosmos3/aste/rc3.d/S99astd start
/opt/OpenSSH/bin/ssh -f aste-3c /cosmos3/aste/nro/rc3.d/S99astnrod start
sleep 20
FACILITY=1 /cosmos3/aste/rc3.d/S99astref1
Starting SYNTHE_A local controler.
FACILITY=2 /cosmos3/aste/rc3.d/S99astref2
Starting SYNTHE_B local controler.
FACILITY=3 /cosmos3/aste/rc3.d/S99astrxt
Starting RXT local controler.
FACILITY=5 /cosmos3/aste/rc3.d/S99astmmc
Starting MMC local controler.
FACILITY=3 /cosmos3/aste/rc3.d/S99astnrorxtd
Starting RXTD daemon.
/cosmos3/aste/nro/rc3.d/S99mmc1
Starting: mmc1_server
/cosmos3/aste/nro/rc3.d/S99sisrxmon
***** aste-v1c ***** [#l0719177]
/opt/OpenSSH/bin/ssh -f aste-v1c /cosmos3/aste/rc3.d/S99astd start
/opt/OpenSSH/bin/ssh -f aste-v1c /cosmos3/aste/nro/rc3.d/S99astnrod start
sleep 20
/cosmos3/aste/nro/rc3.d/S98shmserv
/cosmos3/aste/nro/rc3.d/S99astwebcamd
/cosmos3/aste/nro/sbin/infra_shm_server: Shared memory already allocated
webcam daemon start.
[1] 11565
/cosmos3/aste/nro/rc3.d/S99shmgen
/cosmos3/aste/nro/rc3.d/S99shmcabin
Starting: shm_cabin
cycle power: off
cycle power: on
run main program
[1] 11636
/cosmos3/aste/nro/rc3.d/S99shmshelter
���� ��#��'��$
����
��"������!
220 23 LT-3000 version  3.13  MEIKYO ELECTRIC CO.,LTD. 2003-2004
221 00:09:EE:00:41:15

00:LT>
status=*0000#POS
*00:POS=000,100,100,000 [#q2fbbadc]
00:LT>
Switch status OK
***** aste-c1c ***** [#xdf99b98]
/opt/OpenSSH/bin/ssh -f aste-c1c /cosmos3/aste/rc3.d/S99astd start
���� ��#��'��$
����
��"������!
220 23 LT-3000 version  3.13  MEIKYO ELECTRIC CO.,LTD. 2003-2004
221 00:09:EE:00:41:15

00:LT>
status=*0000#POS
*00:POS=100,100,100,000 [#u946f287]
00:LT>
Switch status OK
/opt/OpenSSH/bin/ssh -f aste-c1c /cosmos3/aste/nro/rc3.d/S99astnrod start
sleep 20
/cosmos3/aste/nro/rc3.d/S99antcadac
Starting cadac6_loop
/cosmos3/aste/nro/rc3.d/S99antcadac21
/cosmos3/aste/nro/rc3.d/S99shmconfroom
/cosmos3/aste/nro/rc3.d/S99weather
/cosmos3/aste/nro/rc3.d/S99shmgenlog
���� ��#��'��$
����
��"������!
220 23 LT-3000 version  3.13  MEIKYO ELECTRIC CO.,LTD. 2003-2004
221 00:09:EE:00:41:13

00:LT>
status=*0000#POS
*00:POS=100,000,100,100 [#b34b5a35]
00:LT>
Switch status OK
/cosmos3/aste/nro/rc3.d/S99shmcabinlog
/cosmos3/aste/nro/rc3.d/S99shmenvlog
���� ��#��'��$
����
��"������!
220 23 LT-3000 version  3.13  MEIKYO ELECTRIC CO.,LTD. 2003-2004
221 00:09:EE:00:41:13

00:LT>
status=*0000#POS
*00:POS=100,100,100,100 [#c1b3bc07]
00:LT>
Switch status OK
***** aste-1c ***** [#df64bee1]
/opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/rc3.d/S99astd start
/opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/nro/rc3.d/S99astnrod start
sleep 20
/cosmos3/aste/nro/rc3.d/S99shmweatherlog
/cosmos3/aste/nro/rc3.d/S99shmmisclog
/cosmos3/aste/nro/rc3.d/S99shmtipper
/cosmos3/aste/nro/rc3.d/S99shmvkput
sleeping for 30 seconds
FACILITY=0 /cosmos3/aste/rc3.d/S99astant
ANTENNA_G local controler starting.
FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvm
astpcvm starting.
FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvs
astpcvs starting.
/cosmos3/aste/nro/rc3.d/S98asp_shm
Starting: asp_shm_server
/cosmos3/aste/nro/rc3.d/S99antstat
Starting: antstat
/cosmos3/aste/nro/rc3.d/S99aspirin
Starting: aspirin
/cosmos3/aste/nro/rc3.d/S99asp_log
Starting: asp-log
/cosmos3/aste/nro/rc3.d/S99asp_watchdog
Starting asp-watchdog
/cosmos3/aste/nro/rc3.d/S99antcadac2asp
Starting cadac2asp
/cosmos3/aste/nro/rc3.d/S99autoweather
Starting: autoweather
/cosmos3/aste/nro/rc3.d/S99wind2shmasp
Starting: wind2shmasp
/cosmos3/aste/nro/rc3.d/S99wind2asp
Starting: wind2asp
starting shm_vkput
/cosmos3/aste/nro/rc3.d/S99antcadac22
���� ��#��'��$
����
��"������!
220 23 LT-3000 version  3.13  MEIKYO ELECTRIC CO.,LTD. 2003-2004
221 00:09:EE:00:41:13

00:LT>
status=*0000#POS
*00:POS=100,100,000,100 [#qddfdef1]
00:LT>
Switch status OK
���� ��#��'��$
����
��"������!
220 23 LT-3000 version  3.13  MEIKYO ELECTRIC CO.,LTD. 2003-2004
221 00:09:EE:00:41:13

00:LT>
status=*0000#POS
*00:POS=100,100,100,100 [#i264c720]
00:LT>
Switch status OK
/cosmos3/aste/nro/rc3.d/S99antcadac22_log
/cosmos3/aste/nro/rc3.d/S99antcadac21_log
shmget: No such file or directory
connect: Connection timed out
k2701: open_connection error
Cannot open k2701 at daq1 port 1394
2014/07/31 02:59:59 ERROR   shm_pressure : serial_read( 1 chars ) timeout
2014/07/31 02:59:59 ERROR   shm_pressure : Read error (serial_read2)
2014/07/31 02:59:59 ERROR   shm_pressure : Failed to read data
2014/07/31 02:59:59 WARNING  shm_pressure : pressure read retry
}}

-Sequence of command:'restart_cosmos_all' (Guessed from message)
--Stopping tasks
---aste-2c  (Obs. container; cos3serv,infoserv,NTP,DASBOX)
---aste-1s  (Obs. container; disk_c,disk_h)
---aste-3c  (Obs. container; MMC,RXT)
---aste-v1c (???; Env?)
---aste-c1c (Cabin; aste-cabin1c)
---aste-1c  (Obs. container; pcv,autoweather)
--Starting tasks
---aste-2c 
---aste-1s 
---aste-3c 
---aste-v1c
---aste-c1c
---aste-1c ; failed with err?

-tasks of aste-1c
--stopping phase;
 /opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/nro/rc3.d/S99astnrod stop
 /opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/rc3.d/S99astd stop
 /cosmos3/aste/nro/rc3.d/S98asp_shm
 Stopping asp_shm_server 9142 9139
 /cosmos3/aste/nro/rc3.d/S99antstat
 /cosmos3/aste/nro/rc3.d/S99aspirin
 /cosmos3/aste/nro/rc3.d/S99asp_log
 /cosmos3/aste/nro/rc3.d/S99asp_watchdog
 /cosmos3/aste/nro/rc3.d/S99antcadac2asp
 /cosmos3/aste/nro/rc3.d/S99autoweather
 Stopping autoweather 9254 9216
 /cosmos3/aste/nro/rc3.d/S99wind2shmasp
 /cosmos3/aste/nro/rc3.d/S99wind2asp
 FACILITY=0 /cosmos3/aste/rc3.d/S99astant
 Stopping ANTENNA_G local controler.
 FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvm
 Stopping astpcvm
 FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvs
 Stopping astpcvs
--In starting section,
 /opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/rc3.d/S99astd start
 /opt/OpenSSH/bin/ssh -f aste-1c /cosmos3/aste/nro/rc3.d/S99astnrod start
 /cosmos3/aste/nro/rc3.d/S99shmweatherlog
 /cosmos3/aste/nro/rc3.d/S99shmmisclog
 /cosmos3/aste/nro/rc3.d/S99shmtipper
 /cosmos3/aste/nro/rc3.d/S99shmvkput
 FACILITY=0 /cosmos3/aste/rc3.d/S99astant
 ANTENNA_G local controler starting.
 FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvm
 astpcvm starting.
 FACILITY=0 /cosmos3/aste/rc3.d/S99astpcvs
 astpcvs starting.
 /cosmos3/aste/nro/rc3.d/S98asp_shm
 Starting: asp_shm_server
 /cosmos3/aste/nro/rc3.d/S99antstat
 Starting: antstat
 /cosmos3/aste/nro/rc3.d/S99aspirin
 Starting: aspirin
 /cosmos3/aste/nro/rc3.d/S99asp_log
 Starting: asp-log
 /cosmos3/aste/nro/rc3.d/S99asp_watchdog
 Starting asp-watchdog
 /cosmos3/aste/nro/rc3.d/S99antcadac2asp
 Starting cadac2asp
 /cosmos3/aste/nro/rc3.d/S99autoweather
 Starting: autoweather
 /cosmos3/aste/nro/rc3.d/S99wind2shmasp
 Starting: wind2shmasp
 /cosmos3/aste/nro/rc3.d/S99wind2asp
 Starting: wind2asp
 starting shm_vkput
 /cosmos3/aste/nro/rc3.d/S99antcadac22
 /cosmos3/aste/nro/rc3.d/S99antcadac22_log
 /cosmos3/aste/nro/rc3.d/S99antcadac21_log 
 shmget: No such file or directory
 connect: Connection timed out
 k2701: open_connection error
 Cannot open k2701 at daq1 port 1394
 2014/07/31 02:59:59 ERROR   shm_pressure : serial_read( 1 chars ) timeout
 2014/07/31 02:59:59 ERROR   shm_pressure : Read error (serial_read2)
 2014/07/31 02:59:59 ERROR   shm_pressure : Failed to read data
 2014/07/31 02:59:59 WARNING  shm_pressure : pressure read retry
--Compared between stopping and starting section,all stopped tasks at aste-1c were tried to start.