Ftp versus webservices

[swamp]

Best way to sincronize/transfer files is rsync:
rsync - Wikipedia, the free encyclopedia

^^^^ WPGS.

Actually transferring files is easy; it's the fail/retry bit that is hard. A good system will get back on track on its own without any extra effort, once the disk space/cron tab/file permission/network problem has been solved.

[Sysman]

If you're worried about the cost of IBM MQ, there's a few decent free alternatives, like Apache ActiveMQ ™ -- Index

I was wondering about that. Getting the budget for IBM MQ yourself is one thing, but if you have external customers they might get stroppy about being forced into it.

[NotAllThere]

Hi All,

Does anyone have any experience of web services as a method of transfering data between systems?

At present we use ftp mainly for this which works well as it is only done once per day.

What are the benefits of using webservices with an ultimate view of going down a full soa route?...

I work with SAP and use web services quite often. The main benefit is the ease of interfacing, without having to worry about what the "other" system is. It also facilitates re-use - anything that can consume a webservice can easily be made to connect - without changing the service at all.

Creating a web service in SAP is handled by wizards and a little config. I then send the WSDL files to the person developing the consumer of the service.

Consuming a web service involves more wizards and config. After that, it's a matter of a few calls to the method of a proxy to the webservice.

The main disadvantage is speed. Web services are some 10x slower than other means of getting data in and out of SAP. It's also not that good, as others have mentioned, for large data volumes - though having said that, we do have one interface that generates the SOAP, then gzips it before sending.

[Freamon]

Cheers Freamon

At present I have no need for real time delivery - an overnight transfer of customer details is the main aim

Therefore to address your points
1) no guaranteed delivery - true but it is easier to if you have 1 sftp file per day to track whether it has been sent or arrived.
2) built in retry-on-failure capability - see 1 can be manually re-sent if needed
3) automated clean up of the directory where the files land - this can be a script - we do it now.
4) capability for asynchronous delivery where the sender or recipient is temporarily down or busy - again with 1 ftp per day if either sending or recieveing system is not available at the point it can be manually re-sent
5) automatic adjustment for network capacity and compensation for network unreliability - true

You can solve those problems with a Message Oriented Middleware product like MQ - MQ is part of the IBM Websphere offering - which sounds expensive!!

Cheers!

Fair enough etc. Obviously MQ is overkill if you only have a requirement to transfer one file per night from A to B, but for an enterprise with lots of similar requirements and no capacity to babysit / manually resend files every day, it has a lot of benefits.

If you're worried about the cost of IBM MQ, there's a few decent free alternatives, like Apache ActiveMQ ™ -- Index

[petergriffin]

Best way to sincronize/transfer files is rsync:
rsync - Wikipedia, the free encyclopedia

[eek]

Cheers Freamon



You can solve those problems with a Message Oriented Middleware product like MQ - MQ is part of the IBM Websphere offering - which sounds expensive!!

Cheers!

MSMQ misses a few nice features but is free on a Windows server. Biztalk would be the appropriate solution if you want an all singing all dancing solution but for basic queue management MSMQ would be fine in allowing you to ensure that a step is completed.

[original PM]

Cheers Freamon

At present I have no need for real time delivery - an overnight transfer of customer details is the main aim

Therefore to address your points
1) no guaranteed delivery - true but it is easier to if you have 1 sftp file per day to track whether it has been sent or arrived.
2) built in retry-on-failure capability - see 1 can be manually re-sent if needed
3) automated clean up of the directory where the files land - this can be a script - we do it now.
4) capability for asynchronous delivery where the sender or recipient is temporarily down or busy - again with 1 ftp per day if either sending or recieveing system is not available at the point it can be manually re-sent
5) automatic adjustment for network capacity and compensation for network unreliability - true

You can solve those problems with a Message Oriented Middleware product like MQ - MQ is part of the IBM Websphere offering - which sounds expensive!!

Cheers!

[Freamon]

The downfall of FTP (or SFTP/SCP) is that there's no guaranteed delivery, built in retry-on-failure capability, automated clean up of the directory where the files land, capability for asynchronous delivery where the sender or recipient is temporarily down or busy, automatic adjustment for network capacity and compensation for network unreliability etc. You can solve those problems with a Message Oriented Middleware product like MQ.

[original PM]

Thanks all.

At present we have no need for real time transactions - however there are a few projects in the pipeline which will need it and we are probably going down the Biztalk route for that.

We currently do a batch transfer of customer details from 1 system to another and this is done overnight via ftp (may even by sFTP tbh).

The "return me a list of XYZ widget and price" is a good example of what we are trying to do - however given that that product list will rarely change from day to day is it feasible to just transfer the data overnight in to database tables in the other app because then rather than having continual interfaces between the 2 systems you have a clear distinction between the two system - obviously the close coupling of two or more systems will mean that if one system goes down then all systems are unavailable or can web services deal with this?

Cheers!

[dangly]

Let me stick my head above the parapet - I'm a BizTalk guy...

As other's have said, it depends on what you trying to do. Webservices are normally good if you are looking at transmitting small-ish messages - think a single record in a batch file, rather than whole batch files - in real-time, i.e. a customer record is updated. If you couple that with the BizTalk pub/sub mechanism, you only need to publish once and multiple subscribers get a copy of the message to do with as they see fit.

I'm working with a client at the moment who is using BizTalk to process real-time pricing updates (c.1 - 50 records) and nightly pricing reconciliation's (tens of thousands of records). Both are generated by their environment as flat-files and put onto MQSeries Queues. BizTalk reads the messages off the queues, 'disassembles' them into an Xml structure (everything in BizTalk is treated as Xml) and publishes them into the Message Box. We have a number of internal systems and external customers who subscribe to the updates and BizTalk transforms the messages into the destination format (we have EDI, custom Xml and flat-file recipients). The BizTalk process is exactly the same for both messages - one just has a lot more records than the other. Given that messages are processed in parallel, we can receive multiple updates at the same time.

We are planning a move to webservices for the real-time pricing updates because they need to be pushed out as soon as they happen on the ERP platform - at the moment, they can take anything upto 30 minutes to be generated on a scheduled job. The nightly reconciliation file will continue to be created as-is and sent over MQSeries as this needs to be generated at a set time every night.

BizTalk is a bit of a beast to get your head around - I've been doing nothing else since 2004 and I'm still learning...

Happy to provide more info if needed.

N.

[DimPrawn]

How big is the data?

Webservices tend to timeout transferring large amounts of data, unless you chunk the data into pieces.

BizTalk adds lots of complexity and it ideal when there are lots of systems involved all requiring different data formats (so lots of transformations).

[cybersquatter]

As with all technology, it depends on what you want to do with it - it's just a means to an end. Why are you considering a switch? If you're talking about batch jobs then ftp/sftp is as good as any. Web services are typically more for the stateless request/response ("return me a list of XYZ widget and price") than for batch jobs.

Not too sure about BizTalk, every time I try and educate myself about it I get more confused. I think the general idea is that it takes data from system A, does some transformation on it, and spits it out to system B with whatever guarantees about reliability/whatever that you wish to have.

[Sysman]

I don't know anything about BizTalk, but sftp or scp are the way to go instead of vanilla ftp. Vanilla ftp doesn't have encryption.

[original PM]

Hi All,

Does anyone have any experience of web services as a method of transfering data between systems?

At present we use ftp mainly for this which works well as it is only done once per day.

What are the benefits of using webservices with an ultimate view of going down a full soa route?

And where does Biztalk fit into this picture.

Any help most appreciated!

Cheers