ALTER DATABASE OPEN
Wed Feb 07 01:59:08 2018
Errors in file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_ora_30542.trc (incident=1232241):
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 30486'
Incident details in: /dbapps/oracle/diag/rdbms/r12t1/R12T1/incident/incdir_1232241/R12T1_ora_30542_i1232241.trc
Wed Feb 07 01:59:38 2018
Killing enqueue blocker (pid=30486) on resource CF-00000000-00000000 by (pid=30542)
by killing session 4401.1
Killing enqueue blocker (pid=30486) on resource CF-00000000-00000000 by (pid=30542)
by terminating the process
USER (ospid: 30542): terminating the instance due to error 2103
Wed Feb 07 01:59:38 2018
System state dump requested by (instance=1, osid=30542), summary=[abnormal instance termination].
System State dumped to trace file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_diag_30478_20180207015938.trc
Dumping diagnostic data in directory=[cdmp_20180207015938], requested by (instance=1, osid=30542), summary=[abnormal instance termination].
Termination issued to instance processes. Waiting for the processes to exit
Wed Feb 07 01:59:49 2018
Instance termination failed to kill one or more processes
Instance terminated by USER, pid = 30542
Errors in file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_dbw0_11551.trc:
ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
ORA-01110: data file 1: '/dba0A/oradata/R12T1/system01.dbf'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8
Additional information: 6315
Errors in file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_dbw0_11551.trc:
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
ORA-01110: data file 2: '/dba0A/oradata/R12T1/system02.dbf'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8
Additional information: 6315
Errors in file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_dbw0_11551.trc:
ORA-01157: cannot identify/lock data file 3 - see DBWR trace file
ORA-01110: data file 3: '/dba0A/oradata/R12T1/system03.dbf'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8
Additional information: 6315
Errors in file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_dbw0_11551.trc:
ORA-01157: cannot identify/lock data file 4 - see DBWR trace file
ORA-01110: data file 4: '/dba0A/oradata/R12T1/system04.dbf'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8
Additional information: 6315
It looks like the DB needed to be restored from backup. However I was curious what was the reason it went down. Examining the alert log from a few days back I saw this line.
Tue Feb 06 17:04:06 2018
KCF: read, write or open error, block=0x108b71 online=1
file=1034 '/dba0B/oradata/R12T1/APPS_TS_TX_DATA108.dbf'
error=27072 txt: 'Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1084273
Additional information: -1'
Errors in file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_dbw0_6231.trc:
Errors in file /dbapps/oracle/diag/rdbms/r12t1/R12T1/trace/R12T1_dbw0_6231.trc:
ORA-63999: data file suffered media failure
ORA-01114: IO error writing block to file 1034 (block # 1084273)
ORA-01110: data file 1034: '/dba0B/oradata/R12T1/APPS_TS_TX_DATA108.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1084273
Additional information: -1
DBW0 (ospid: 6231): terminating the instance due to error 63999
I checked it by
! ORCL_11204:server.domain:/dbapps/oracle/diag/rdbms/ORCL/ORCL/trace> cd /dba0B/oradata/ORCL/
! ORCL_11204:server.domain:/dba0B/oradata/ORCL> vi something.txt
! ORCL_11204:server.domain:/dba0B/oradata/ORCL> touch something
touch: cannot touch `something': Read-only file system
I told the SA to check /dba0B and make it read only.
After getting the confirmation he fixed it. I was able to startup the DB normally.
No comments:
Post a Comment