Reply to: .NET file locks

[AtW]

I didn't produce that bit though.

No you didn't produce it.

I am chuffed at guessing that the thing was actually doing copying/deletion rather than moving (renaming)

[ASB]

So I was right then?

Using temp dir (as provided by "helpful" .NET function) and then moving file is crazy practice precisely because of this sort of situation, what if the file is 100 GB?

That's why SKA always creates temp file in the same directory where it should be in the first place, just with .tmp extension - when it's done, file is closed and moved to it's proper final name - very nice operation on NTFS.

I didn't produce that bit though. As I said previously I'm not in control of that bit (though it seems I am now; ho hum - be careful what you wish for and all that!).

[AtW]

I understand your argument now.

AtW is awarded 51 Zip points

[Zippy]

I understand your argument now.

AtW is awarded 51 Zip points

[AtW]

Even if you can move, I think that is actually still a copy and delete operation so you will still encounter the odd exception. My feeling is still that living with it is the least expensive option.

When on the same partition move is a rename, very quick atomic (AFAIK) operation on NTFS.

Copying indeed can be expensive, however in this case cost isn't an issue - the issue is that most stupid move commands will not first copy file into TEMPORARY filename on the target drive and then RENAME it to finish the operation: that's how well designed software should operate, this way it's easy to know when file has been FULLY copied/moved without any filehandle issues or if box get rebooted in between it's easy to check which files were fully copied and which didn't.

It's a rookie error to get temp file that will be on OS partition and then "move" it to final destination which can be anywhere far away.

Just something I've learnt whilst dealing with hundreds of terabytes of SKA data...

Where is Xeno when you need him?

I'd say 51 geek points are in order here...

[Zippy]

Even if you can move, I think that is actually still a copy and delete operation so you will still encounter the odd exception. My feeling is still that living with it is the least expensive option.
AtW probably knows more about this though.

[AtW]

But: I think I have a solution. After a bit of pleading I am now allowed to modify P1 since what is actually happening is it is creating the file in the temporary directory and moving it to the real directory. Guess what: different partitions so it is really copying.

So I was right then?

Using temp dir (as provided by "helpful" .NET function) and then moving file is crazy practice precisely because of this sort of situation, what if the file is 100 GB?

That's why SKA always creates temp file in the same directory where it should be in the first place, just with .tmp extension - when it's done, file is closed and moved to it's proper final name - very nice operation on NTFS.

[ASB]

1. Move (rename) in the same directory - if fails for whatever reason wait a bit and retry.
2. Only THEN copy - this will obtain lock on a file and P1 may fail if it tries to write to that file whilst you copy.

Simples.

AtW,

But that just moves the problem in that there is the same issue on the rename in the current directory (the rename may fail with an exception due to the fact it is still locked by P1). Granted I may reasonably be able to assume that is is more likely the failure is because it is locked but there is no guarantee of that.

But: I think I have a solution. After a bit of pleading I am now allowed to modify P1 since what is actually happening is it is creating the file in the temporary directory and moving it to the real directory. Guess what: different partitions so it is really copying.

Thus I can change P1 to move to .tmp and then rename to .xxx.

If the rename is truly atomic problem dealt with, if not then the recovery currently in place will resolve it.

Of course the test I need to do first is to ascertain the relative costs of the exception 1 in (say) 100 and the extra rename every time. I suspect (but don't know until I write some test code) that it might be better to live with the exception.

[AtW]

- P2 is actually simply copying the file to a remote server for processing
- P2 is supposed to get it there as quickly as possible

1. Move (rename) in the same directory - if fails for whatever reason wait a bit and retry.
2. Only THEN copy - this will obtain lock on a file and P1 may fail if it tries to write to that file whilst you copy.

Simples.

[ASB]

Quick search indicates there isn't - just handling the exception, which it sounds like you're doing. The reasons make sense - between checking for the lock and opening the file, it could become locked (in theory).

Can you check the timestamp of the file, and only process if > 5 secs old for example?

Had done a lot of searching, and I came to the same conclusion. I did consider the timestamp, but the problems with that are:-

- P2 is actually simply copying the file to a remote server for processing
- P2 is supposed to get it there as quickly as possible
- Any mechanism of using API to check lock is only masking the issue since it could get locked in the intervening time [not actually possible in this case - at least in theory]
- I am not sure of the granularity of timestamps in any event (though it shouldn't take much effort to work out)

P2 is fully restartable, as is P1 in fact. My beef really is just that I have to trap the exception - and it's not specific enough to be sure. This does mean that I might - for example - happen to retry when it's a network failure on the copy - and I can't tell from the exception.

Live with it seems to be the right answer.

[AtW]

Would you want to rely on that? Move it to a different system, with different partitions/drives, or have process 1 and 2 on different physical machines and it won't work anymore.

It will work so long as the following principle applies: movement of file (which is a simple atomic rename operation in this case) should be within the same directory where it was created (say with .tmp extension) and written. If file is moved between partitions then it should be copied into temporary file and then renamed at the target disk - I am pretty sure in NTFS (at least recent versions) renaming (or moving within same disk) is atomic.

Is it perfect? No - but easy and pretty reliable so long as you use good file system like NTFS.

Either way in this case process 1 can't be changed, there ought to be a function in win32 to check if file is already open or some other lock is obtained on it.

[VectraMan]

normal moves should be instant and pretty much atomic on NTFS

Would you want to rely on that? Move it to a different system, with different partitions/drives, or have process 1 and 2 on different physical machines and it won't work anymore.

Seems to me detecting the failure, waiting, and having another go is the right approach. If the move is atomic, it won't ever fail, so you don't lose anything. If you need something more instantaneous, then you need to find a better mechanism for communicating between two processes than the existance of a file.

[AtW]

Is this "moving" by process 1 done on the same partition? If not, then it's actually copy/delete_old operation which would indeed have locks, normal moves should be instant and pretty much atomic on NTFS - at least in my experience: SKA does move a fair few files with other processes reading them.

Even if it's same partition it may well be that programmer of process 1 did not use proper move command and just did copy/delete.

Check this out, it may help - http://vbnet.mvps.org/code/fileapi/createfile_inuse.htm

[mudskipper]

Quick search indicates there isn't - just handling the exception, which it sounds like you're doing. The reasons make sense - between checking for the lock and opening the file, it could become locked (in theory).

Can you check the timestamp of the file, and only process if > 5 secs old for example?

[ASB]

I think the problem is that process 2 is trying to access the file during the move/copy process, so the lock is still on.

Correct. It's actually doing a move not a copy so it is at least pretty quick (in any event the files are pretty small). The exceptions are occasional and it just produces an IO exception so it does a quick sleep and then retries. It's just that exceptions are expensive and I would prefer to be able to tell more precisely that it failed because of a lock error.

It's just that it's not really an exception - in that it is a predictable fail condition.