Really odd FTP issue

[WTFH]

I've just been asked about a really odd FTP issue.
A text file is created on one server and then FTP'd to another.
The next day a new file is created with the same name and FTP'd across.

Here is the problem: The new file does not replace the old one, but seems to be overwriting the data in it.
That wouldn't be a problem if the new file was always bigger than the old one, but if the new file is smaller in size than the old one, then you see the e.g. 50 rows of the new file, and row 51 onwards is actually row 51 onwards of the old file.

Any thoughts?

[eek]

I've just been asked about a really odd FTP issue.
A text file is created on one server and then FTP'd to another.
The next day a new file is created with the same name and FTP'd across.

Here is the problem: The new file does not replace the old one, but seems to be overwriting the data in it.
That wouldn't be a problem if the new file was always bigger than the old one, but if the new file is smaller in size than the old one, then you see the e.g. 50 rows of the new file, and row 51 onwards is actually row 51 onwards of the old file.

Any thoughts?

You've got a dodgy FTP client which is opening the existing file rather than deleting it and starting afresh...

[woody1]

As I recall, with UNIX, FTP put overwrites the file (same behaviour as cp/copy). Which OS is this, and is it built-in FTP or 3rd party?

[WTFH]

As I recall, with UNIX, FTP put overwrites the file (same behaviour as cp/copy). Which OS is this, and is it built-in FTP or 3rd party?

It’s going from Windows to UNIX, using built-in FTP.

[hobnob]

Make sure that you're using binary mode to transfer the file. If you use ASCII mode, Windows and Unix handle line breaks differently (CR+LF on Windows, LF only on Unix). I haven't seen it cause this particular issue, but I have seen it cause corrupted files before.

[woody1]

It’s going from Windows to UNIX, using built-in FTP.

Weird. Never seen this before, nor could I find this issue with Google.

As an experiment, it might be worth changing mode (binary/ascii) to see if this changes the behaviour.

-----------

I'd also use od (-a or -c) to examine the file on the UNIX side to see if there's anything odd around row 50.

[WTFH]

Weird. Never seen this before, nor could I find this issue with Google.

As an experiment, it might be worth changing mode (binary/ascii) to see if this changes the behaviour.

-----------

I'd also use od (-a or -c) to examine the file on the UNIX side to see if there's anything odd around row 50.

No, nothing special around row 50 - we've tried with other file sizes and it is just overwriting the new data as far as it needs to in the file.
The people that the data is going to have now discovered that the new file has its original header and the new header as well.
I wonder if we can send a Delete before we send the next file, or maybe move/rename the previous file when we send the new one.

[woody1]

No, nothing special around row 50 - we've tried with other file sizes and it is just overwriting the new data as far as it needs to in the file.
The people that the data is going to have now discovered that the new file has its original header and the new header as well.

Even more bizarre. I can't imagine it being an FTP server configuration option because who would want this behaviour.

Which UNIX is it? Sun, HP, Linux?

[b0redom]

Not seen this before. I wonder if it's the windows client 'restarting' the file transfer. TBH though, knowingly overwriting a file is bad practice operationally. If you think you might need it, make a backup, if you don't remove it as soon as it's done. Either of those are trivial using cron.

[NickFitz]

No, nothing special around row 50 - we've tried with other file sizes and it is just overwriting the new data as far as it needs to in the file.

It sounds like the destination file is being opened for writing, but never closed. That would leave it hanging and presumably the OS would eventually close the open file handle but without any awareness of what had been done to it. This could, in theory, leave the content on disk having been overwritten but the file descriptor not having been told the new length of the file. Then the file would have the new content at the start but have the same length as the old file from the point of view of whatever reads it.

The people that the data is going to have now discovered that the new file has its original header and the new header as well.

This also makes it sound like the file modification is happening but being left hanging. But by this point, I'd be demanding hex dumps of individual disk sectors so I could see what was really going on

I wonder if we can send a Delete before we send the next file, or maybe move/rename the previous file when we send the new one.

That might work but if it's what I suggest, then fixing the sending FTP client to properly close the file (or the receiving one to do so if it's their fault) would resolve the issue by making things work properly, which is better than a clumsy and haphazard "fix" that seems to work and does… for now