[Rets-dev] RE: RETS 2 Vocabulary intent

Thu May 18 13:58:36 CDT 2006

Steve,

I guess I wasn't clear earlier.

in my mind, this vocabulary issue has absolutely everything to do with 
DELIMITED payloads.

As I see it, the only time the non-searchable fields are ever used are in 
the context of a 'select'. And the 'select' is only used for DELIMITED 
payload requests.

Does anyone see another use for non-searchable names that I am missing?

As for the duplicate elements in Vocabulary vs. DataDictionary, don't think 
of them as database tables that need to be normalized but as different 
'views' of the same base internal information set. The purpose of the 
Vocabulary is to supply in a single request the minimum infomation needed to 
perform a search for an XML response.

Jeff

>From: "Steve Clarke" <Steve.clarke at firstamericanmls.com>
>To: "Paul Stusiak" <pstusiak at falcontechnologies.com>
>CC: rets-dev at rets.org
>Subject: RE: RETS 2 Vocabulary intent WAS: [Rets-dev] RETS 2 Service 
>Documentposted to www.ftc2.com
>Date: Thu, 18 May 2006 14:18:22 -0400
>
>
> >> ...but the Vocabulary concept is derived from
> >> what those who attended the many meetings
> >> discussed and agreed upon.
>
>Paul,
>
>I hope you're not implying that I did not participate in this discussion
>and that I'm now coming in at the last minute.  Also, in my mind, this
>vocabulary issue has absolutely nothing to do with DELIMITED payloads or
>any other specific functional area of the spec.  This is a fundamental
>metadata issue that has ramifications on the abilities for a server and
>client to effectively communicate metadata changes to each other.
>
>I participated in this area of the spec from the beginning (in fact, I
>introduced this concept to the group).  The whole subject began with an
>extremely lengthy debate on RETS-DEV over whether the dictionary should
>be segmented vertically to allow for role-based granularity.  The whole
>premise of doing this was to (be able to) minimize metadata changes
>required for RETS clients that support many roles on a single client.
>
>I participated in this process at the Las Vegas meeting, where I think
>it actually took hold with the skeptics because this additional
>segmentation does not add any limitation or significant complexity to
>vendors that either do not need to support a large number of roles or
>prefer to implement their roles via completely separate dictionaries.  I
>think that it was in Las Vegas where we actually decided to call this
>"vocabulary" in the first place.
>
>At that point, I felt comfortable that the concept was being adopted by
>the group.  I think that it's clear that I am not the only one that was
>surprised by the difference between what I thought was going into the
>spec and what actually is in the spec.  I apologize that I did not
>continue to monitor this concept closely to make sure that it was not
>being broken by subsequent adjustments.  However, I certainly did catch
>it prior to the vote.
>
>So now we need to figure out how to fix it.  If the spec passed the
>vote, then I will draft change proposals to be voted on.  If the spec
>did not pass, the let's just fix it for the next vote.  As I said, I do
>not believe that the fix is very complicated.  I also do not think that
>the fix (dropping R0043) will bloat the vocabularies.  In fact, the
>vocabularies will likely shrink more by the removal of the redundant
>attributes.  So far I have not heard any strong argument as to why R0043
>must stand.  We already know that clients must be prepared to handle
>error related to vocabulary fields that are not searchable by a person's
>role.  We also already have the dictionary attribute called
>"IsSearchable."  In fact, all arguments to keep R0043 seem to come with
>an admission that something else is missing from the metadata.  I would
>seriously hope that that's not the case.
>
>Seems to me the fix is really simple:
>
>1) Remove redundant LookupName and DataType from vocabulary altogether.
>2) Remove R0043 from spec.
>
>smc
>-----Original Message-----
>From: Paul Stusiak [mailto:pstusiak at falcontechnologies.com]
>Sent: Thursday, May 18, 2006 1:45 PM
>To: Sergio.Del.Rio at t4bi.com; Sergio Del Rio
>Cc: jbrush at ronintech.org; Steve Clarke; rets-dev at rets.org
>Subject: RETS 2 Vocabulary intent WAS: [Rets-dev] RETS 2 Service
>Document posted to www.ftc2.com
>
>First, let's separate the Subject into a thread related to its content.
>
>Second, while I recognize that many of you are very busy, this has been
>out in
>the document for many months. It shouldn't be a surprise to you.
>
>Third, over the period of discussion, many ideas were expressed, some
>were
>incorporated into the RETS2 Service document. Some were not.
>
>Fourth, let's clear up the word query. From the context of the RETS2
>Service
>document and RQL, the word query refers to the criteria provided in the
>request
>to return payloads. There may be uses where Query refers to entire SQL
>statements, but in the RETS2 context, Query refers ONLY to the criteria.
>
>This topic started in workgroup discussions during November, had some
>ideas
>expressed in a very very long thread and was further discussed a number
>of
>times: at the December meeting in Las Vegas, on metadata calls in
>January,
>February and March 2006, all after the original "reduction" dev list
>discussion.
>See the minutes for Jan. 12th and the agenda material for the 2/14
>meeting.
>During the course of this discussion, the Vocabulary concept was
>expressed,
>starting in the December meeting.
>
>This was voted and adopted by the whole group on April 13th, 2006 in
>Washington, DC.
>
>Vocabulary and Payloads are an attempt to provide the separation of
>concerns.
>
>The intent of Vocabulary is to make it simpler to build systems by
>providing
>information to Requestors on what should go into the criteria to return
>documents. Please see page 6-13. The field aliases allows the Provider
>to give
>the Requestor the field names in a form that is appropriate to the
>context of
>the Request - Real Estate names for an agent, REPI names for an
>appraiser,
>transaction names for a transaction coordinator, MISMO names for a
>lender. The
>grouping information exists to permit Requestors to understand how the
>elements
>should be presented and which are required. While a case could be made
>that
>there is redundant information in this metadata, one of the requests
>from
>Requestor vendors (clients in RETS1 speak) is to make it simpler to use
>the
>metadata. Placing the information in two places will put a small burden
>on
>Providers while delivering a larger benefit to Requestors - the
>information is
>in one place. It provides one and only one purpose: to indicate those
>fields
>that can be used to form the criteria to retrieve documents for a
>Resource.
>Provider vendors may choose to change this list based on role or person.
>This is
>an implementation issue. What is important from the standard and
>interoperability perspective is to provide some guidance to Requestors
>that this
>behavior may occur.
>
>If, after reading this section, you believe that different text or
>additional
>requirements would make this clearer, please let me know and I will add
>it.
>
>I foolishly attempted to make it clear what the behavior of a system
>should be
>under the conditions of a Provider who reduces the metadata based on
>person or
>role. This was not defined clearly in RETS1 and has lead to
>interoperability
>issues. I believe that the requirement that was added, to force
>WellKnown names
>to always be present regardless of the role or person, would clarify
>this issue.
>It did not. Jeff Brush and Steve Clarke have provided wording that
>clarifies my
>intent in adding this to the specification. Those changes have been
>added.
>
>If you are suggesting that we need additional metadata for DELIMITED to
>tell
>clients what they can "see" that is one thing, but the Vocabulary
>concept is
>derived from what those who attended the many meetings discussed and
>agreed
>upon. It seems a little unfair to those who have started development to
>be
>suggesting a change of this type at this time after it has been adopted.
>
>I respect the opinions of those who have contributed to this thread and
>their
>right to change their minds, but this has been discussed at length, has
>been
>vetted by both Provider and Requestor vendors, has been voted on and is
>in
>development. If we need to add to the standard to meet cases that the
>standard
>does not currently address, then the community will add them.
>
>Paul
>
>Quoting Sergio Del Rio <sergio at mris.com>:
>
> > First of all, it sounds like you are agreeing that we should not have
> > LookupName and DataType in the Vocabulary.  Is this a correct
>interpretation
> > of your comments?
> >
> > Now, on the Searchable issue: Why do we need to state that all fields
>in a
> > Vocabulary must be searchable?  The intent of the Vocabulary was to
>allow
> > subsets of fields to be sent to the Client to indicate which fields
>they
> > were allowed to see (most likely based on server side roles).  The
>Data
> > Dictionary could then either be: a) A more global data dictionary for
> > servers that are not concerned about showing metadata about restricted
> > fields; b) A role specific data dictionary (hopefully not unique per
> > individual, but rather by dictionary name which would be unique) for
>servers
> > that do not want to show metadata about restricted fields.
> >
> > At least this is what I remember from all those very lengthy e-mail
>threads
> > and it sounds like Steve is of the same opinion.
> >
> > So, forcing the fields in Vocabulary to be searchable, actually breaks
>this
> > original intent because now we have no way of sending a Vocabulary
>that has
> > a basic set of fields that are viewable for a user.
> >
> > Regards,
> > Sergio Del Rio
> > Templates 4 Business Inc.
> >
> >
> > -----Original Message-----
> > From: rets-dev-bounces at rets.org [mailto:rets-dev-bounces at rets.org] On
>Behalf
> > Of Jeff Brush
> > Sent: May 18, 2006 8:16 AM
> > To: Steve.clarke at firstamericanmls.com;
>pstusiak at falcontechnologies.com;
> > rets-dev at rets.org
> > Subject: RE: [Rets-dev] RETS 2 Service Document posted to www.ftc2.com
> >
> >
> > At the risk of 'twisting' and defeating 'original' purposes let me
>explain
> > some of the rational behind R0043- the requirement that vocabularies
>must be
> >
> > searchable.
> >
> > The expectation for RETS2 is that most requests will be for XML
>documents.
> > This was true to the extent that the RETS community even voted to
>exclude
> > COMPACT from RETS2.
> >
> > In the context of XML, the only field names that are relavent are
>thoses
> > used in the query clause. Does it make sense to at least quadruple (or
>even
> > 10x) the size of all vocabulary document to support non-searchable
>fields
> > that will not be referenced.
> >
> > In the case of DELIMITED payloads, DELIMITED data was added back into
>the
> > spec at the request of client-side users. The primary purpose was to
>allow
> > data replication. Because MLSs historically have a difficult time
>mapping
> > names other than system names to their data, we didn't feel the need
>at this
> >
> > time to implement other vocabularies that have yet to be defined. (We
>hope
> > to identify a more optimal replication method in the future.)
> >
> > A little perspective is sometimes helpful.
> >
> > Jeff Brush
> > Chief Architect
> > Ronin Technologies
> >
> >
> >
> > >From: Steve Clarke [mailto:Steve.clarke at firstamericanmls.com]
> > >Sent: May 18, 2006 6:28 AM
> > >To: Sergio Del Rio; Paul Stusiak; rets-dev at rets.org
> > >Subject: RE: [Rets-dev] RETS 2 Service Document posted to
>www.ftc2.com
> > >
> > >Yea.  I thought we called that a "vocabulary" too.  But in the
>current
> > >draft spec, that meaning (of vocabulary) has been twisted by R0043
>which
> > >now attaches an inherent functional limitation (must be searchable)
>on
> > >the vocabulary fields.  That change now implies that we are
> > >substantially missing something from the metadata and it also creates
>a
> > >redundancy with the IsSearchable attribute in the dictionary.  In my
> > >mind, the fact that we even have an IsSearchable attribute in the
> > >dictionary validates the original intent of a vocabulary to be
>exactly
> > >what you are asking about.  Certainly that's what I thought too.  But
> > >R0043 kinda defeats the original purpose.
> > >
> > >Smc
> > >
> > >-----Original Message-----
> > >From: rets-dev-bounces at rets.org [mailto:rets-dev-bounces at rets.org] On
> > >Behalf Of Sergio Del Rio
> > >Sent: Wednesday, May 17, 2006 11:31 PM
> > >To: 'Paul Stusiak'; rets-dev at rets.org
> > >Subject: RE: [Rets-dev] RETS 2 Service Document posted to
>www.ftc2.com
> > >
> > >OK, I must be tired.  I thought it was there at some point, but now I
> > >can't
> > >see it.  What happened to the metadata tag concept that was going to
> > >reduce
> > >the amount of different metadata that we were going to have.  Did we
> > >replace
> > >this simply with different Vocabulary or Data Dictionary versions?
> > >
> > >Regards,
> > >Sergio Del Rio
> > >Templates 4 Business Inc.
> > >
> > >
> > >-----Original Message-----
> > >From: rets-dev-bounces at rets.org [mailto:rets-dev-bounces at rets.org] On
> > >Behalf
> > >Of Paul Stusiak
> > >Sent: May 16, 2006 3:02 PM
> > >To: rets-dev at rets.org
> > >Subject: [Rets-dev] RETS 2 Service Document posted to www.ftc2.com
> > >
> > >Changes for FC5
> > >
> > >1. Remove section 2.4
> > >
> > >2. Add requirement
> > >
> > >R0148	A Provider SHOULD return a fault when an invalid Field
>Name is
> > >used
> > >in a
> > >Query by the Requestor.
> > >
> > >and modify the text preceeding to clarify the behavior of Vocabulary
> > >under
> > >security and system reductions.
> > >
> > >3. Correct select statement for the Delimted ObjectType to use either
> > >the
> > >WellKnown Vocabulary field names or the DataDictionary field names.
> > >
> > >4. Update R0138 from presented to present
> > >
> > >5. Update R0139, adding the preamble "If DisplayGroups are present in
> > >the
> > >Vocabulary document," to the beginning of the document.
> > >
> > >6. Added a reference section pointing to the RQL Query Language
> > >documents.
> > >They
> > >are currently only available on the mirror site and not on rets.org.
> > >
> > >The document will be bookmarked and an index added after the final
>vote.
> > >
> > >It is now posted to www.ftc2.com
> > >-----------
> > >Paul Stusiak,
> > >President,
> > >Falcon Technologies Corp.
> > >_______________________________________________
> > >Rets-dev mailing list
> > >Rets-dev at rets.org
> > >http://lists.rets.org/mailman/listinfo/rets-dev
> > >
> > >
> > >
> > >_______________________________________________
> > >Rets-dev mailing list
> > >Rets-dev at rets.org
> > >http://lists.rets.org/mailman/listinfo/rets-dev
> > >
> > >
> > >_______________________________________________
> > >Rets-dev mailing list
> > >Rets-dev at rets.org
> > >http://lists.rets.org/mailman/listinfo/rets-dev
> >
> >
> > _______________________________________________
> > Rets-dev mailing list
> > Rets-dev at rets.org
> > http://lists.rets.org/mailman/listinfo/rets-dev
> >
> >
> >
>
>
>-----------
>Paul Stusiak,
>President,
>Falcon Technologies Corp.
>_______________________________________________
>Rets-dev mailing list
>Rets-dev at rets.org
>http://lists.rets.org/mailman/listinfo/rets-dev