Question

[Unix] How to wait for a file to be available

Hello unix experts,

1) I need to wait for a file to exist,
2) Once the file begins to exist, I need to wait again for it to become ready (fully copied from RAM to disk)
3) then use the file to kickstart other jobs

The file is very huge and it takes ~1 hour for it to get copied to disk(step 2 above)

My question is -
Is there a way to know the difference between :
1) simple 'existence of file' and
2) 'file is stable, no other process is writing to it

Answer 1

I started with some code for waiting for a file $metal :-
```
sub wait_for_fill {
my $metal = 'fill_trex12g_DMDV.gds';
while(1) {
if (! -e "$metal") {
print "\nWAITING for file : $metal\n";
sleep(60);
}else {
return;
}
}
}
```

The thing throwing me off w/ this code is - I’m not sure whether perl treats the $metal file as ready while the file is still getting copied to disk.

Anybody tweaked this before and have a working code ?

Answer 2

hmm Interesting problem...have you found any solution, please share?
I am not sure but I guess OS will itself add a lock to the file till it is not copied completely and ready to use. There is a command in linux called fuser to check if any other process is using this file and after that add a flock till your operation is complete.

Answer 3

Checking to see if the file is writable should work, but a more common thing have seen is to write a lock file at the start of the copy and delete the lock file at the end of the copy. Then the existence of the lock file is what you check for.

Answer 4

if [[ -e $FILE ]]
then
# check the mtime of the file against the system time
# the difference should be greater than a quiet time for the file
# the specifics vary from dialect to dialect
# time is held as an integer, so the math is straightforward
fi

Answer 5

@ganeshie8 i am ready

Answer 6

since im in linux environment, can we mess wid stat or inode or something similar ?

for now i have implemented the task by checking for the datastamp of file every 10 minutes, until it matches the older datastamp...

just wondering if there is a more efficient way...

Answer 7

The problem is you are doing it in the OS, rather than inside a program... usually a program itself handles these things.

Answer 8

operating system ? i didnt get it @e.mccormick , the file generation program runs on a server system and the file can be ready anytime - I dont have much control over the file or the server program; I can oly wait for the file to exist on my partition and become stable for using it as input to other programs.

Answer 9

Exactly. Exactly. If you write a program, you get to use whatever methods you want. When you are just using the system utilities and other programs, you get limited by what they do.

Answer 10

What are you using to download the file?

Answer 11

Testing whether a file exists is done with "if [[ -e $FILE ]]". You can stat the file and get it's mtime (Modification TIME). Subtract that from the current time, and check that against a quiescence period (one minute = 60).

Time is kept as an signed integer from the epoch (Jan 1, 1970 00:00:00 UTC), so time math is simple to do.

Answer 12

that was how i had implemented @rsmith6559 and its working fine :)

so affter all the research, looks like there is no **direct** way of knowing the difference between :

1) simple 'existence of file' and
2) 'file is stable, no other process is writing to it

Answer 13

@e.mccormick the server is also in local intranet. the program runs on the server and in the end, it creates this file on my partition, and starts dumping the data from memory to this file... (its a direct write system call to a partition i would believe, as all machines are under one intranet.)

guess i should be satisfied wid the datastamp checking method. ive been using this for last couple of days, its working pretty good... :) i may have to increase the quiescence period in case it cribs in future... thanks a lot everyone appreciate very much :)

Answer 14

If you find the time thing undesirable, you could check for file locks on the file, or if the process that writes the file has exited. lsof could be handy, but this strikes me as more of a C program than shell level.

Answer 15

@rsmith6559 which is what I was pointing out... hehe.