Ftp versus webservices

**Sysman** · 22 July 2011, 12:09

I don't know anything about BizTalk, but sftp or scp are the way to go instead of vanilla ftp. Vanilla ftp doesn't have encryption.

**cybersquatter** · 22 July 2011, 13:38

As with all technology, it depends on what you want to do with it - it's just a means to an end. Why are you considering a switch? If you're talking about batch jobs then ftp/sftp is as good as any. Web services are typically more for the stateless request/response ("return me a list of XYZ widget and price") than for batch jobs.

Not too sure about BizTalk, every time I try and educate myself about it I get more confused. I think the general idea is that it takes data from system A, does some transformation on it, and spits it out to system B with whatever guarantees about reliability/whatever that you wish to have.

**DimPrawn** · 22 July 2011, 13:44

How big is the data?

Webservices tend to timeout transferring large amounts of data, unless you chunk the data into pieces.

BizTalk adds lots of complexity and it ideal when there are lots of systems involved all requiring different data formats (so lots of transformations).

**dangly** · 22 July 2011, 15:40

Let me stick my head above the parapet - I'm a BizTalk guy...

As other's have said, it depends on what you trying to do. Webservices are normally good if you are looking at transmitting small-ish messages - think a single record in a batch file, rather than whole batch files - in real-time, i.e. a customer record is updated. If you couple that with the BizTalk pub/sub mechanism, you only need to publish once and multiple subscribers get a copy of the message to do with as they see fit.

I'm working with a client at the moment who is using BizTalk to process real-time pricing updates (c.1 - 50 records) and nightly pricing reconciliation's (tens of thousands of records). Both are generated by their environment as flat-files and put onto MQSeries Queues. BizTalk reads the messages off the queues, 'disassembles' them into an Xml structure (everything in BizTalk is treated as Xml) and publishes them into the Message Box. We have a number of internal systems and external customers who subscribe to the updates and BizTalk transforms the messages into the destination format (we have EDI, custom Xml and flat-file recipients). The BizTalk process is exactly the same for both messages - one just has a lot more records than the other. Given that messages are processed in parallel, we can receive multiple updates at the same time.

We are planning a move to webservices for the real-time pricing updates because they need to be pushed out as soon as they happen on the ERP platform - at the moment, they can take anything upto 30 minutes to be generated on a scheduled job. The nightly reconciliation file will continue to be created as-is and sent over MQSeries as this needs to be generated at a set time every night.

BizTalk is a bit of a beast to get your head around - I've been doing nothing else since 2004 and I'm still learning...

Happy to provide more info if needed.

N.

**original PM** · 22 July 2011, 16:35

Thanks all.

At present we have no need for real time transactions - however there are a few projects in the pipeline which will need it and we are probably going down the Biztalk route for that.

We currently do a batch transfer of customer details from 1 system to another and this is done overnight via ftp (may even by sFTP tbh).

The "return me a list of XYZ widget and price" is a good example of what we are trying to do - however given that that product list will rarely change from day to day is it feasible to just transfer the data overnight in to database tables in the other app because then rather than having continual interfaces between the 2 systems you have a clear distinction between the two system - obviously the close coupling of two or more systems will mean that if one system goes down then all systems are unavailable or can web services deal with this?

Cheers!

**Freamon** · 22 July 2011, 23:00

The downfall of FTP (or SFTP/SCP) is that there's no guaranteed delivery, built in retry-on-failure capability, automated clean up of the directory where the files land, capability for asynchronous delivery where the sender or recipient is temporarily down or busy, automatic adjustment for network capacity and compensation for network unreliability etc. You can solve those problems with a Message Oriented Middleware product like MQ.

**original PM** · 24 July 2011, 12:58

Cheers Freamon

At present I have no need for real time delivery - an overnight transfer of customer details is the main aim

Therefore to address your points
1) no guaranteed delivery - true but it is easier to if you have 1 sftp file per day to track whether it has been sent or arrived.
2) built in retry-on-failure capability - see 1 can be manually re-sent if needed
3) automated clean up of the directory where the files land - this can be a script - we do it now.
4) capability for asynchronous delivery where the sender or recipient is temporarily down or busy - again with 1 ftp per day if either sending or recieveing system is not available at the point it can be manually re-sent
5) automatic adjustment for network capacity and compensation for network unreliability - true

You can solve those problems with a Message Oriented Middleware product like MQ - MQ is part of the IBM Websphere offering - which sounds expensive!!

Cheers!

**eek** · 25 July 2011, 06:18

Originally posted by original PM View Post

Cheers Freamon

You can solve those problems with a Message Oriented Middleware product like MQ - MQ is part of the IBM Websphere offering - which sounds expensive!!

Cheers!

MSMQ misses a few nice features but is free on a Windows server. Biztalk would be the appropriate solution if you want an all singing all dancing solution but for basic queue management MSMQ would be fine in allowing you to ensure that a step is completed.

**petergriffin** · 25 July 2011, 17:40

Best way to sincronize/transfer files is rsync:
rsync - Wikipedia, the free encyclopedia

Ftp versus webservices