.NET file locks

[AtW]

- P2 is actually simply copying the file to a remote server for processing
- P2 is supposed to get it there as quickly as possible

1. Move (rename) in the same directory - if fails for whatever reason wait a bit and retry.
2. Only THEN copy - this will obtain lock on a file and P1 may fail if it tries to write to that file whilst you copy.

Simples.

 

[ASB]

1. Move (rename) in the same directory - if fails for whatever reason wait a bit and retry.
2. Only THEN copy - this will obtain lock on a file and P1 may fail if it tries to write to that file whilst you copy.

Simples.

AtW,

But that just moves the problem in that there is the same issue on the rename in the current directory (the rename may fail with an exception due to the fact it is still locked by P1). Granted I may reasonably be able to assume that is is more likely the failure is because it is locked but there is no guarantee of that.

But: I think I have a solution. After a bit of pleading I am now allowed to modify P1 since what is actually happening is it is creating the file in the temporary directory and moving it to the real directory. Guess what: different partitions so it is really copying.

Thus I can change P1 to move to .tmp and then rename to .xxx.

If the rename is truly atomic problem dealt with, if not then the recovery currently in place will resolve it.

Of course the test I need to do first is to ascertain the relative costs of the exception 1 in (say) 100 and the extra rename every time. I suspect (but don't know until I write some test code) that it might be better to live with the exception.

 

[AtW]

But: I think I have a solution. After a bit of pleading I am now allowed to modify P1 since what is actually happening is it is creating the file in the temporary directory and moving it to the real directory. Guess what: different partitions so it is really copying.

So I was right then?

Using temp dir (as provided by "helpful" .NET function) and then moving file is crazy practice precisely because of this sort of situation, what if the file is 100 GB?

That's why SKA always creates temp file in the same directory where it should be in the first place, just with .tmp extension - when it's done, file is closed and moved to it's proper final name - very nice operation on NTFS.

 

[Zippy]

Even if you can move, I think that is actually still a copy and delete operation so you will still encounter the odd exception. My feeling is still that living with it is the least expensive option.
AtW probably knows more about this though.

 

[AtW]

Even if you can move, I think that is actually still a copy and delete operation so you will still encounter the odd exception. My feeling is still that living with it is the least expensive option.

When on the same partition move is a rename, very quick atomic (AFAIK) operation on NTFS.

Copying indeed can be expensive, however in this case cost isn't an issue - the issue is that most stupid move commands will not first copy file into TEMPORARY filename on the target drive and then RENAME it to finish the operation: that's how well designed software should operate, this way it's easy to know when file has been FULLY copied/moved without any filehandle issues or if box get rebooted in between it's easy to check which files were fully copied and which didn't.

It's a rookie error to get temp file that will be on OS partition and then "move" it to final destination which can be anywhere far away.

Just something I've learnt whilst dealing with hundreds of terabytes of SKA data...

Where is Xeno when you need him?

I'd say 51 geek points are in order here...

 

[Zippy]

I understand your argument now.

AtW is awarded 51 Zip points

 

[AtW]

I understand your argument now.

AtW is awarded 51 Zip points

 

[ASB]

So I was right then?

Using temp dir (as provided by "helpful" .NET function) and then moving file is crazy practice precisely because of this sort of situation, what if the file is 100 GB?

That's why SKA always creates temp file in the same directory where it should be in the first place, just with .tmp extension - when it's done, file is closed and moved to it's proper final name - very nice operation on NTFS.

I didn't produce that bit though. As I said previously I'm not in control of that bit (though it seems I am now; ho hum - be careful what you wish for and all that!).

 

[AtW]

I didn't produce that bit though.

No you didn't produce it.

I am chuffed at guessing that the thing was actually doing copying/deletion rather than moving (renaming)