[Rets-dev] RETS manifest

[Jeff Brush]

Please read 4 messages down. Or read the MTOM spec:

http://www.w3.org/TR/soap12-mtom/ - , specifically Section '2.1 
Introduction', first paragraph, last sentence.


>From: "Wantao Zhou" <Wantao.Zhou at firstamericanmls.com>
>To: <jbrush at ronintech.org>, <pstusiak at falcontechnologies.com>
>CC: rets-dev at rets.org
>Subject: RE: [Rets-dev] RETS manifest
>Date: Wed, 26 Jul 2006 12:33:46 -0400
>
>I think we all agree that we will be using MTOM to deliver large
>messages. The idea to verify is whether it has to be binary in order to
>take advantage of MTOM. The link doesn't say it has to be binary. My
>interpretation is that it even suggested to use MTOM to deliver large
>XMLs as long as IXmlSerializable is implemented. It just doesn't seem
>right that you have to have binary data in the attachment.
>
>Wantao
>
>-----Original Message-----
>From: Jeff Brush [mailto:jeffbrush at hotmail.com]
>Sent: Wednesday, July 26, 2006 12:11 AM
>To: Wantao Zhou; pstusiak at falcontechnologies.com
>Cc: rets-dev at rets.org
>Subject: RE: [Rets-dev] RETS manifest
>
>Paul asked me to explain the repercussions of what I found in the
>link/post
>- Sending large SOAP envelopes -
>http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedback
>id=ac2db95b-bc39-48cc-bc0e-f357322e4b93
>
>Mark Fussell, the WSE Program Manager at Microsoft, responded to a
>question
>about streaming of large SOAP messages. He stated:
>
>"However with WSE 3.0 (as with WSE 2.0) is still XmlDocument class based
>
>internally and so all messages will be loaded into memory so you can
>never
>achieve complete streaming end to end. (Other than when written to the
>network stream as described above for MTOM). "
>
>I interpret this to mean, in the current design of WSE 3.0, the SOAP
>envelope, which would include the SOAP headers, SOAP body, and in our
>case,
>the RETS Manifest are all loaded into a XmlDocument object in memory
>before
>being sent. And this is not true with MTOM'ed elements.
>
>Mark says "[MTOM enables] a file to be directly written to the network
>stream by implementing IXmlSerializable." I do not interpret this to
>mean
>simplly using IXmlSerializable makes it streamable. If it was that easy,
>
>streaming would not be the issue.
>
>He ends his comments with:
>
>"I suggest that you have a look at the use of MTOM for large message, if
>you
>can move to WSE 3.0".
>
>I think that describes the situation we are facing with RETS2.
>
>
>Jeff
>
>
>----Original Message Follows----
>From: "Jeff Brush" <jeffbrush at hotmail.com>
>Reply-To: jbrush at ronintech.org
>To: Wantao.Zhou at firstamericanmls.com, pstusiak at falcontechnologies.com
>CC: rets-dev at rets.org
>Subject: RE: [Rets-dev] RETS manifest
>Date: Wed, 26 Jul 2006 01:41:45 -0400
>
>This was my fear:
>
>Sending large SOAP envelopes -
>http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedback
>id=ac2db95b-bc39-48cc-bc0e-f357322e4b93
>
>My interpretation of this is the WSE toolkits internally represent the
>whole
>SOAP envelope as an in-memory XMLDocument except the MTOM attachments.
>
>BTW- I've gotten the impression WCF fixes this.
>
>Jeff
>
>----Original Message Follows----
>From: "Jeff Brush" <jeffbrush at hotmail.com>
>Reply-To: jbrush at ronintech.org
>To: Wantao.Zhou at firstamericanmls.com, jbrush at ronintech.org,
>pstusiak at falcontechnologies.com
>CC: rets-dev at rets.org
>Subject: RE: [Rets-dev] RETS manifest
>Date: Wed, 26 Jul 2006 01:07:09 -0400
>
> >From the MTOM spec: "Optimization is available only for element content
>that is in a canonical lexical representation of the xs:base64Binary
>data
>type."
>
>So a base64Binary data type signals the optimizer to put raw,
>unencoded data bytes into a multipart attachment. This is what MTOM does
>-
>raw un-base64 encoded data in an attachment and an XOP tag taking the
>base64Binary data's place. Period. (Some toolkits seem to support
>hexBinary
>as well.)
>
>Is your position is that regular XML content can be streamed using
>classes
>implementing System.Xml.IXmlSerializable.[WriteXML | ReadXML]. Perhaps
>because of "[WebMethod(BufferResponse = false)]"? I'll look into that
>some
>more... If someone could code a simple example that generates a large
>(10-30
>Mb) mock XML document and see if the data streams or is all loaded into
>memory first, that could prove it.
>
>Also for Java developers, java data mappings to base64Binary elements:
>
>In xFire, the java classes/datatypes, DataSource, DataHandler, and
>byte[],
>are automatically mapped by the framework. DataSource and DataHandler
>are
>streams.
>
>In Axis2, DataHandler and a base64 encoded String are mapped.
>
>In JWSDP 2.0, DataHandler, Image, Source and a byte[] are mapped.
>
>This all makes it very transparent for an Java object to bind to the
>data
>stream.
>
>This may all be moot in that the latest generation of toolkits seem to
>claim
>to support general streaming of the SOAP body via StAX for java. For MS,
>
>WSE3 doesn't but WCF is suppost to.
>
>What we could do is change the Manifest to allow either a <Content> or
>an
><XmlContent> tag.  <XmlContent> contains <xs:any namespace="##other">.
>And
><Content> is a base64Binary data type that may contain an XML payload if
>its
>mime type is "xml/text". This would allow a Provider to use the solution
>
>they like best.
>
>Jeff
>
>----Original Message Follows----
>From: "Wantao Zhou" <Wantao.Zhou at firstamericanmls.com>
>To: <jbrush at ronintech.org>,<pstusiak at falcontechnologies.com>
>CC: <rets-dev at rets.org>
>Subject: RE: [Rets-dev] RETS manifest
>Date: Tue, 25 Jul 2006 17:26:15 -0400
>
>I think what the first document is trying to demonstrate is how to
>stream binary data, and therefore by definition the result is binary.
>However this doesn't mean that you have to have binary data in order to
>be streamed. If the XmlSchemaProvider specifies a different schema, for
>example a schema of a "regular" XML, and the proper WriteXml and ReadXml
>methods are provided, then MTOM will then stream the data properly.
>
>Wantao
>
>-----Original Message-----
>From: Jeff Brush [mailto:jeffbrush at hotmail.com]
>Sent: Tuesday, July 25, 2006 1:45 PM
>To: Wantao Zhou; pstusiak at falcontechnologies.com
>Cc: rets-dev at rets.org
>Subject: RE: [Rets-dev] RETS manifest
>
>I am not sure the links I thought I sent out ever made it so here they
>are
>again. The 1st one is the most relevant. The others load the byte array
>into
>memory.
>
>How to: Stream Large Amounts of Data from a Web Service -
>http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wse3.0/
>html/3e9ce678-2502-4847-9f13-5173896c9db5.asp
>
>How to: Send and Receive Large Amounts of Data to and from a Web Service
>-
>http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wse3.0/
>html/b4b19453-e4e4-4056-906d-72504ed8c0df.asp
>
>How to: Enable a Web Service to Send and Receive Large Amounts of Data -
>
>http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wse3.0/
>html/4344d43e-ceb4-43a9-8f8c-6a3f89f786bd.asp
>
>----Original Message Follows----
>From: "Wantao Zhou" <Wantao.Zhou at firstamericanmls.com>
>To: "Paul Stusiak" <pstusiak at falcontechnologies.com>
>CC: rets-dev at rets.org
>Subject: RE: [Rets-dev] RETS manifest
>Date: Wed, 19 Jul 2006 14:31:59 -0400
>
>Thanks for the explanation.
>
>The manifest doesn't seem to be part of the MTOM standard. If that's the
>case, how would MTOM be able to "look at the manifest"?
>
>Specifically, the way MTOM is supported in WSE3 on the .Net platform is
>almost purely by configuration. The web method would return a parameter,
>which based on the current spec, would be the manifest. However, because
>the content element is defined as base64binary in the schema, it will be
>declared as byte[] in .Net. This is not desirable for XML document.
>
>To solve this problem, we can either declare multiple types of manifest
>document or multiple types of content element within the same manifest
>document.
>
>Wantao
>
>-----Original Message-----
>From: Paul Stusiak [mailto:pstusiak at falcontechnologies.com]
>Sent: Wednesday, July 19, 2006 10:56 AM
>To: Wantao Zhou
>Cc: rets-dev at rets.org
>Subject: Re: [Rets-dev] RETS manifest
>
>The manifest provides MTOM with the information that it needs to
>determine the encoding technique used.
>
>An example:
>
><Content type="text/xml" >xml document in-line</Content>
>
>MTOM looks at the manifest and determines that the content is of type
>text/xml and that additional processing (assuming that document or field
>
>encryption is not specified anywhere - this is a future addition to
>RETS2) is not necessary in this case and packages it up as a multi-part
>mime attachment.
>
>A second example:
>
><Content type="image/jpg" >binary data in-line</Content>
>
>MTOM looks at the manifest and determines that the content is of type
>image/jpg and that additional processing is required and then does the
>necessary BASE64 encoding and attaches it as a multi-part mime
>attachment. At the other end, MTOM reverses the process.
>
>So the answer is no, not all of the information will be Base64 encoded.
>It depends what the type is. There is a case to be made for adding a
>couple of sample documents to RET2 Service which I'll do. Hopefully that
>
>will make this clearer.
>
>When using MTOM, there is no need to deal directly with the BASE64
>encoding. It is part of the plumbing, so to speak.
>
>It also looks like we should clarify and add/modify the requirement that
>
>'SHOULD use MTOM'.
>
>Maybe for mime types text/plain or text/xml it can be SHOULD, MUST use
>MTOM for binary mime types and MUST support non-MTOM mime types
>text/plain or text/xml. We need to think about this a bit. It would be
>simplest to force MTOM attachments, but this may not be the most
>desirable. Comments on this are welcomed.
>
>Paul
>
>Wantao Zhou wrote:
>   >
>   > Hi everyone,
>   >
>   > Can someone clarify the use of manifest in RETS 2? On page 6-3 of
>RC5,
>
>   > the document says:
>   >
>   > RETS2 payloads may be XML documents, binary data such as .jpg or
>.gif
>   > images, or other MIME types
>   >
>   > supported by a Provider.
>   >
>   > The payloads are packaged inside of the RETS Manifest. To preserve
>the
>
>   > logical structure of the
>   >
>   > document, RETS uses a manifest document to describe the structure,
>   > allowing the contents to be
>   >
>   > 610 serialized as necessary.
>   >
>   > This is then followed by the schema of the RETS manifest. The
>manifest
>
>   > contains three parts, the description, count, and the content. The
>   > content is defined as base 64 encoded, regardless the type of data.
>   >
>   > If I understand it correctly, all returned listing data, even if
>   > that's purely textual, will be packed into a non-human readable,
>   > non-XML, based 64 encoded "content" element.
>   >
>   > Is there a good reason why we want to do this? It's reasonable to
>   > encode binary data like images, but not the regular listing data. It
>   > doesn't seem to make implementation easier, or make performance
>better.
>   >
>   > Wantao
>   >
>   >
>------------------------------------------------------------------------
>   >
>   > _______________________________________________
>   > Rets-dev mailing list
>   > Rets-dev at rets.org
>   > http://lists.rets.org/mailman/listinfo/rets-dev
>   >
>_______________________________________________
>Rets-dev mailing list
>Rets-dev at rets.org
>http://lists.rets.org/mailman/listinfo/rets-dev
>
>
>_______________________________________________
>Rets-dev mailing list
>Rets-dev at rets.org
>http://lists.rets.org/mailman/listinfo/rets-dev
>
>
>_______________________________________________
>Rets-dev mailing list
>Rets-dev at rets.org
>http://lists.rets.org/mailman/listinfo/rets-dev