1.
  alert文件中有ora-01114和ora-29701错误。
  查看解释：
/home/oracle:$oerr ora 1114
01114, 00000, "IO error writing block to file %s (block # %s)"
// *Cause:  The device on which the file resides is probably offline. If the
//          file is a temporary file, then it is also possible that the device
//          has run out of space. This could happen because disk space of
//          temporary files is not necessarily allocated at file creation time.
// *Action: Restore access to the device or remove unnecessary files to free
//          up space.
/home/oracle:$oerr ora 29701
29701, 00000, "unable to connect to Cluster Manager"
// *Cause: Connect to CM failed or timed out.
// *Action: Verify that the CM was started.  If the CM was not started,
//          start it and then retry the database startup. If the CM died
//          or is not responding, check the Oracle and CM trace files for
//          errors.


2.
 1114：  临时文件所在目录磁盘没有空间
 29701： 连接CM失败

  由于使用的ASM，进入ASMCMD
  ASMCMD> lsdg             --空间还有很多
  $ df -g                  --$HOME目录还有11g空间

3.
  查看对应trace文件
  *** SESSION ID:(1329.8651) 2013-06-05 15:46:56.561
*** CLIENT ID:(java.lang.Thread) 2013-06-05 15:46:56.561
*** SERVICE NAME:(EPMS) 2013-06-05 15:46:56.561
*** MODULE NAME:(ManageMyUserProfile) 2013-06-05 15:46:56.561
*** ACTION NAME:(saveMyUserProfile) 2013-06-05 15:46:56.561

2013-06-05 15:46:56.551: [ CSSCLNT]clssscConnect: gipcWait failed with 16 (12)
2013-06-05 15:46:56.562: [ CSSCLNT]clsssInitNative: connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_p750b_)) failed, rc 16
kgxgncin: CLSS init failed with status 3
kgxgncin: return status 3 (1311719766 SKGXN not av) from CLSS
kjfmsgr: unable to connect to NM for reg in shared group

*** 2013-06-05 15:46:56.567
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)
----- Error Stack Dump -----
ORA-01114:
ORA-29701:    --这里都是乱码就不粘上了。

查看对应ocssd.log
2013-06-05 15:47:14.289: [GIPCXCPT][1029] gipcmodMuxTransferAccept: internal accept request failed endp 1115f1930, child 11444ad50, ret gipcretAuthFail (22)
2013-06-05 15:47:14.289: [ GIPCMUX][1029] gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ]  error during accept on endp 1115f1930
2013-06-05 15:47:14.290: [GIPCXCPT][1029] gipcmodClscCallback: async request failed req 113dd4d30 [0000000008120825] { gipcSendRequest : addr '', data 1139d2810, len 48, olen 0, parentEndp 113fad5f0, ret gipcretConnectionLost (12), objFlags 0x0, reqFlags 0x224 }, ret gipcretConnectionLost (12)
2013-06-05 15:47:14.290: [GIPCXCPT][1029] gipcmodMuxTransferAccept: internal accept request failed endp 1115f1930, child 113fad5f0, ret gipcretConnectionInvalid (13)
2013-06-05 15:47:14.290: [ GIPCMUX][1029] gipcmodMuxTransferAccept:
 EXCEPTION[ ret gipcretConnectionInvalid (13) ]  error during accept on endp 1115f1930
2013-06-05 15:47:14.334: [    CSSD][1029]clssscSelect: cookie accept request 110d228e0
2013-06-05 15:47:14.334: [    CSSD][1029]clssgmAllocProc: (1140bbc70) allocated
2013-06-05 15:47:14.339: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1140bbc70 - 0,1,2,3,4
2013-06-05 15:47:14.339: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(8120868) proc(1140bbc70) pid(32505882/32505882) version 11:2:1:4, properties: 0,1,2,3,4
2013-06-05 15:47:14.339: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2013-06-05 15:47:14.363: [    CSSD][1029]clssscSelect: cookie accept request 1140bbc70
2013-06-05 15:47:14.363: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1140bbc70
2013-06-05 15:47:14.363: [    CSSD][1029]clssgmRegisterClient: proc(229/1140bbc70), client(1/113e2c590)
2013-06-05 15:47:14.369: [    CSSD][1029]clssgmRegisterShared: grp DBEPMS, mbr 1, type 1
2013-06-05 15:47:14.369: [    CSSD][1029]clssgmQueueShare: (113ffdcb0) target global grock DBEPMS member 1 type
1 queued from client (113e2c590), global grock DBEPMS, refcount 129
2013-06-05 15:47:14.369: [    CSSD][1029]clssgmRegisterShared: global grock DBEPMS member 1 share type 1, refcount 129
2013-06-05 15:47:14.394: [    CSSD][1029]clssscSelect: cookie accept request 1140bbc70
2013-06-05 15:47:14.394: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1140bbc70

4.  search for metalink
I/O Errors in Alert log with ORA-29701, with "gipcWait failed with 16" in trace [ID 1496329.1]
Details
  Cause 1. ocssd is not able to process all the requests efficiently enough so some get rejected -    due to bug 11069614
Cause 2. ocssd or ocssd threads are not scheduled with high enough priority
  Solaris & HP-UX Only
  AIX specific
  Cause 3. ocssd log has "gipcretAuthFail (22)"
  Cause 4. ocssd log has "gipcmodClsaAuthStart" failure with "Too many open files" in SIHA
  Cause 5. Oracle Restart setup is incorrect
  Some potential workarounds
 这里类型Cause 3.

Cause 3. ocssd log has "gipcretAuthFail (22)"

Example:

gipcretAuthFail (22) indicates "general security authorization failure".

This could occur for multiple reasons:
* if filesystem is full and there is no space to create file under auth directory. Please check if there is sufficient space in CRS_HOME.
* Also this issue could occur if /var/tmp/.oracle socket is deleted (/tmp/.oracle on some platforms) . Please check on this too.

5.

  检查对应目录，磁盘空间都很充足。

  /var/temp/.oracle  是有的。

6.

   错误之后的日志，都正常的，我觉得可能是突然间的高峰导致的。