Redirect file sporadically not generated, permissions OK

[daniel.kopko]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' 
-DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/l\
ocale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib  
-D_GNU_SOURCE -DRECYCLES_PIDS  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 
-fexceptions -fstack-prot\
ector --param=ssp-buffer-size=4 -m64 -mtune=generic -fwrapv
uname output: Linux njblxpx39d 2.6.32-358.23.2.el6.x86_64 #1 SMP Sat Sep 14 
05:32:37 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu

Bash Version: 4.1
Patch Level: 2
Release Status: release

Description:
*Sometimes* a redirected file is not generated, even though directory 
permissions are fine.

Repeat-By:

THE RELEVANT CODE SECTION

                log_notice "DANDEBUG Storing pidfile... ${PIDFILE}"
                ps -p ${NEWPID} -o pid= -o args= >${PIDFILE}
                PSRET=$?
                log_notice "DANDEBUG PSRET: ${PSRET}"
                if [[ -f "${PIDFILE}" ]]
                then
                        log_notice "DANDEBUG PIDFILE exists."
                        log_notice "DANDEBUG Contents of pidfile..."
                        cat ${PIDFILE}
                else
                        log_error "PIDFILE doesn't exist."
                fi

Each morning I have two cronjobs (dscache_copsmaster, dscache_copsslave) which 
each launch an application through a common "appbase.sh" application script.  
The way this works is that each defines a PROC_COMMAND_START and then sources 
appbase.sh which will give commands like "start", "stop", "status", such that 
things like the following can be called: "./dscache_copsmaster.sh start", 
"./dscache_copsmaster.sh stop".  Starting an application in this way will 
create a requisite run directory and put a pidfile there for reference by a 
later call to stop.  Every morning, when the instance called dscache_copsmaster 
is started, it complains that the PIDFILE doesn't exist.  However, for 
dscache_copsslave, the correct behavior occurs and there is no complaint. The 
failure output is as follows:


20141909 03:00:01.037   NOTICE  DANDEBUG Storing pidfile... 
/apps/home/nye_pxd/logs/today/dscache/dscache_copsmaster/dscache_copsmaster.pid
20141909 03:00:01.074   NOTICE  DANDEBUG PSRET: 0
20141909 03:00:01.075   ERROR   PIDFILE doesn't exist.
20141909 03:00:06.077   NOTICE  dscache_copsmaster started (pid: 46465).


However, dscache_copsslave (and all other applications managed through the same 
script) always succeeds and here is its successful output:
20141909 03:00:01.037   NOTICE  DANDEBUG Storing pidfile... 
/apps/home/nye_pxd/logs/today/dscache/dscache_copsslave/dscache_copsslave.pid
20141909 03:00:01.070   NOTICE  DANDEBUG PSRET: 0
20141909 03:00:01.070   NOTICE  DANDEBUG PIDFILE exists.
20141909 03:00:01.071   NOTICE  DANDEBUG Contents of pidfile...
 46464 /apps/home/nye_pxd/dscache/bin/dscache 
/apps/home/nye_pxd/dscache/config/dscache_copsslave.shadow.amer.xml
20141909 03:00:06.074   NOTICE  dscache_copsslave started (pid: 46464).


Other relevant bits:
1) This section of the script runs with "set -o nounset" and "set -o pipefail"
2) The directories in which the pidfiles are to be written are created by the 
script in an earlier section, and are writable (confirmed, because the 
application itself outputs a file there immediately on starting).  Here is how 
they are created:
        [[ -d "${RUNDIR}" ]] || mkdir -p ${RUNDIR}
Where RUNDIR for dscache_copsmaster is 
"/apps/home/nye_pxd/logs/today/dscache/dscache_copsmaster" and for 
dscache_copsslave is "/apps/home/nye_pxd/logs/today/dscache/dscache_copsslave".
3) The only differences between the two application launch scripts is as 
follows:
$ diff dscache_copsmaster.sh dscache_copsslave.sh
17c17
< PROC_COMMAND_START="${APPROOT}/bin/dscache 
${APPROOT}/config/dscache_copsmaster.${PX_ENV}.${PX_REGION}.xml"
---
> PROC_COMMAND_START="${APPROOT}/bin/dscache 
> ${APPROOT}/config/dscache_copsslave.${PX_ENV}.${PX_REGION}.xml"
4) The $? of the ps command is confirmed to be 0 through the above log output, 
though even if it failed I would have expected ${PIDFILE} to have been created.
5) The file does not exist even after the script exits, causing problems with 
later requests for "status" and "stop" which depend on its contents.  I could 
see that perhaps the creation had a race with the test for file, but what is 
very odd to me is that the file is not created at all even by when the script 
completes.  I could deal with first touching the file before redirecting to it 
if redirects were known to race with the file test, but the above example seems 
to indicate that redirects sometimes silently entirely fails to create the file.

_______________________________________________

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

_______________________________________________