• Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
  • Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

Ftp versus webservices

Collapse
X
  •  
  • Filter
  • Time
  • Show
Clear All
new posts

    Ftp versus webservices

    Hi All,

    Does anyone have any experience of web services as a method of transfering data between systems?

    At present we use ftp mainly for this which works well as it is only done once per day.

    What are the benefits of using webservices with an ultimate view of going down a full soa route?

    And where does Biztalk fit into this picture.

    Any help most appreciated!

    Cheers

    #2
    I don't know anything about BizTalk, but sftp or scp are the way to go instead of vanilla ftp. Vanilla ftp doesn't have encryption.
    Behold the warranty -- the bold print giveth and the fine print taketh away.

    Comment


      #3
      As with all technology, it depends on what you want to do with it - it's just a means to an end. Why are you considering a switch? If you're talking about batch jobs then ftp/sftp is as good as any. Web services are typically more for the stateless request/response ("return me a list of XYZ widget and price") than for batch jobs.

      Not too sure about BizTalk, every time I try and educate myself about it I get more confused. I think the general idea is that it takes data from system A, does some transformation on it, and spits it out to system B with whatever guarantees about reliability/whatever that you wish to have.

      Comment


        #4
        How big is the data?

        Webservices tend to timeout transferring large amounts of data, unless you chunk the data into pieces.

        BizTalk adds lots of complexity and it ideal when there are lots of systems involved all requiring different data formats (so lots of transformations).

        Comment


          #5
          Let me stick my head above the parapet - I'm a BizTalk guy...

          As other's have said, it depends on what you trying to do. Webservices are normally good if you are looking at transmitting small-ish messages - think a single record in a batch file, rather than whole batch files - in real-time, i.e. a customer record is updated. If you couple that with the BizTalk pub/sub mechanism, you only need to publish once and multiple subscribers get a copy of the message to do with as they see fit.

          I'm working with a client at the moment who is using BizTalk to process real-time pricing updates (c.1 - 50 records) and nightly pricing reconciliation's (tens of thousands of records). Both are generated by their environment as flat-files and put onto MQSeries Queues. BizTalk reads the messages off the queues, 'disassembles' them into an Xml structure (everything in BizTalk is treated as Xml) and publishes them into the Message Box. We have a number of internal systems and external customers who subscribe to the updates and BizTalk transforms the messages into the destination format (we have EDI, custom Xml and flat-file recipients). The BizTalk process is exactly the same for both messages - one just has a lot more records than the other. Given that messages are processed in parallel, we can receive multiple updates at the same time.

          We are planning a move to webservices for the real-time pricing updates because they need to be pushed out as soon as they happen on the ERP platform - at the moment, they can take anything upto 30 minutes to be generated on a scheduled job. The nightly reconciliation file will continue to be created as-is and sent over MQSeries as this needs to be generated at a set time every night.

          BizTalk is a bit of a beast to get your head around - I've been doing nothing else since 2004 and I'm still learning...

          Happy to provide more info if needed.

          N.

          Comment


            #6
            Thanks all.

            At present we have no need for real time transactions - however there are a few projects in the pipeline which will need it and we are probably going down the Biztalk route for that.

            We currently do a batch transfer of customer details from 1 system to another and this is done overnight via ftp (may even by sFTP tbh).

            The "return me a list of XYZ widget and price" is a good example of what we are trying to do - however given that that product list will rarely change from day to day is it feasible to just transfer the data overnight in to database tables in the other app because then rather than having continual interfaces between the 2 systems you have a clear distinction between the two system - obviously the close coupling of two or more systems will mean that if one system goes down then all systems are unavailable or can web services deal with this?

            Cheers!

            Comment


              #7
              The downfall of FTP (or SFTP/SCP) is that there's no guaranteed delivery, built in retry-on-failure capability, automated clean up of the directory where the files land, capability for asynchronous delivery where the sender or recipient is temporarily down or busy, automatic adjustment for network capacity and compensation for network unreliability etc. You can solve those problems with a Message Oriented Middleware product like MQ.
              "A life, Jimmy, you know what that is? It’s the s*** that happens while you’re waiting for moments that never come." -- Lester Freamon

              Comment


                #8
                Cheers Freamon

                At present I have no need for real time delivery - an overnight transfer of customer details is the main aim

                Therefore to address your points
                1) no guaranteed delivery - true but it is easier to if you have 1 sftp file per day to track whether it has been sent or arrived.
                2) built in retry-on-failure capability - see 1 can be manually re-sent if needed
                3) automated clean up of the directory where the files land - this can be a script - we do it now.
                4) capability for asynchronous delivery where the sender or recipient is temporarily down or busy - again with 1 ftp per day if either sending or recieveing system is not available at the point it can be manually re-sent
                5) automatic adjustment for network capacity and compensation for network unreliability - true

                You can solve those problems with a Message Oriented Middleware product like MQ - MQ is part of the IBM Websphere offering - which sounds expensive!!

                Cheers!

                Comment


                  #9
                  Originally posted by original PM View Post
                  Cheers Freamon



                  You can solve those problems with a Message Oriented Middleware product like MQ - MQ is part of the IBM Websphere offering - which sounds expensive!!

                  Cheers!

                  MSMQ misses a few nice features but is free on a Windows server. Biztalk would be the appropriate solution if you want an all singing all dancing solution but for basic queue management MSMQ would be fine in allowing you to ensure that a step is completed.
                  merely at clientco for the entertainment

                  Comment


                    #10
                    Best way to sincronize/transfer files is rsync:
                    rsync - Wikipedia, the free encyclopedia
                    <Insert idea here> will never be adopted because the politicians are in the pockets of the banks!

                    Comment

                    Working...
                    X