[Rets-dev] Roll-up clarifications to some of the threads of the last several days.

[Paul Stusiak]

Since we seem to have had some misconceptions presented as facts, I feel 
that it is important to correct them.

The State of RETS 1

The state of the RETS 1 standard names informed many of the decisions 
for RETS 2. Some of the concerns voiced were that there was no 
consistency between the various MLS data sources and that there was too 
much work needed to create a simple client. The simple client was 
described as a client that needed access to a small number of fields 
that were, hopefully, consistent across all MLS systems. These clients 
may directly serve a human audience - IDX type systems used for Internet 
marketing or they may be used as part of a larger system - Automated 
Valuation or Current Market Analysis system, mapping system "mash-up" or 
as an input to another system - collecting the seller name and contact 
information to feed into the appraisal process or other service order. 
RETS 1 did not meet this need. To access many parts of the information, 
a metadata aware client was required, even some elements that should 
have been well-known proved to be missing from the standard names. In 
many cases, this awareness was gained by having a programmer examine the 
information, contact the support staff to get the meaning of the element 
and then to code against that information. Changes to the information 
returned from the system frequently required changes to the code.

While many people who regularly attend the RETS meetings understand that 
a metadata aware client is relatively straight-forward to implement, 
many of the people who need to get to the RETS data are not easily able 
to build such clients.

Additionally, there was no mechanism in RETS 1 to extend the system, 
using an XML format, to return all the information described in the 
metadata. Some locations provided their own extensions, but there was no 
consistent method to extend the XML with RETS 1.

The second concern voiced, that it was difficult to build clients 
against RETS 1 systems indicated that there were not enough tools. Each 
client started from scratch, implementing the RETS 1 standard. While 
there are some good sources of code available, many developers chose to 
build it themselves. Customization was then applied to each systems that 
the client connected to, hopefully limited to a small number of changes, 
but in practice, the changes seemed to be predicated on the MLS system 
vendor. Some vendors had a consistent implementation of RETS 1 across 
all servers at all MLS sites while others placed that control in the 
hands of the MLS, to determine when and if they should move to a more 
recent version. Field differences also caused differences between sites 
and systems, requiring customization at some level for each site.

Finally, for the purposes of this discussion - there are several other 
concerns that have validity with RETS 1 that I will not discuss, there 
was a concern that the community had spent too much time in creating 
standards that were handled by other, existing standards rather than 
focus on the domain specific concerns. Let existing standards for 
transport, data description and security be handled by other standards 
and weave those together into a new standard.

Some Choices Around RETS2

One possible solution to the problems described for RETS 1 was to use 
more external standards to assist in the development. Most tool vendors 
and platform vendors had invested heavily in Web Services. This 
technology suite also had broad adoption in the enterprise space, 
suggesting that the pool of knowledge would be very broad.

While the WS suite is fairly complete, including well defined discovery 
and security, the level of expertise required to work with this suite 
was more complex than that of RETS 1. Most, if not all of the developers 
who attend RETS meetings and participate on the rets-dev list have 
sufficient expertise to work with this suite of technology and have 
voiced their support. There is a substantial body of developers who will 
not fit into this category who have need to access the information 
provided by RETS. This gap will be bridged by providing tools to further 
abstract the details of RETS2 from these developers. While this is 
important, it is not particularly germane to the discussion of schema.

The choice of Web Services strongly implies the use of XML Schema in the 
delivery of the response and practically mandate the use of XML in that 
response. While legacy and distributed database concerns are addressed 
in the  use of the DELIMITED formats, other users are encouraged to use 
one of the payloads (schema) described by RETS2.

To be very clear, the RETS2 Service document, the document that 
discusses the basic architecture and the 'transport' protocols has been 
ratified and adopted by the RETS community. Part of the adoption 
includes the use of schema as an integral part of the standard and of 
compliance. In this community we are attempting to discuss the details 
of the implementation of the schema.

RETS2 Schemas

Specifically, in RETS2, there are discovery schema that should permit a 
service requestor (client) to determine which resources (payload types) 
are available on a specific service as well as using the wsdl to 
determine (and build stubs) the actual request types that are provided.

There are a large number of different payloads described for RETS2. 
These are not considered an exhaustive list. Changing technology or 
local conditions will generate new payloads that will be described 
outside of those defined by RETS2, but the mechanism to create them is 
present. It is also expected that many MLS sites will require extensions 
to the defined payloads. Again there is a mechanism to extend the 
payloads in a standard way. There is no mechanism for over-riding the 
definitions in RETS2. It is not permitted.

Let me repeat the last couple of sentences so that everyone is clear. 
Again there is a mechanism to extend the payloads in a standard way. 
There is no mechanism for over-riding the definitions in RETS2. It is 
not permitted. While it may appear possible, the validation step will 
break in generated clients when the redefinition is encountered. 
Specifically, over-riding in the same namespace is not a compliant act. 
As part of compliance, as discussed and decided, the validation will be 
against the reference and not the instance.

This has been discussed at length over the past two years. For those of 
just now joining the conversation, many of these discussions occurred at 
the various meetings, including teleconference calls. These were 
announced through the rets-dev mailing list on an open conference line. 
For those of you attempting to re-open these choices, you have rehashed 
the same arguments that were rejected over the last several years. 
You'll have to come up with new arguments to sway the community. 
Reiterating your position is unlikely to change any minds.

The extension mechanism uses the standard XML Schema definition of using 
a separate namespace to create these extensions. Please refer to the 
simple example of extension that was posted prior to the last meeting.

Also note that there is an un-typed payload provided to help those who 
need to see only a small set of the data, the IDX payloads.

For the majority of uses, stronger typing is needed to ensure that the 
data actually has some meaning. Using an area value of 'HUGE' is not 
possible for legal description or for CMA or AVM or regular Appraisal 
purposes never mind trying to sell the property.

I would generally describe the best practices to be, assuming that the 
MLS has decided that the data should be retained, taking the small set 
of information that contains this type of bad information and re-map to 
an extension field of localmls-area-comment or some such name since it 
isn't really an area at all, but a comment about an area. Data in the 
same field that has 'correct' information would pass through using the 
rets standard name. There are several ways to do this remapping, either 
at run-time or more likely, as part of a data mapping exercise. Other 
solutions are possible than what I describe as a best practice. I do 
claim that the method described in a thread earlier this week is not the 
correct solution. Over-riding the definition will break clients.

There are several structural and naming problems with the existing 
schema. While the discussion did not get to these points because of the 
concerns of a few of the attendees. There seems to be an interest from 
many groups in the industry to move to a stronger typing system. While 
we can argue about the relative merits of typing and attempt to get 100% 
of the information from 100% of the locations, there is value in moving 
to stronger typing.

The current state of the schema is as a proposal. At least one 
completely untyped schema has been presented. As has been pointed out in 
private communications, there is a minimum set of typing needed to make 
RETS2 schema useful. For those who need it, within RETS2 there is a 
DELIMITED and ENCODED-DELIMITED formats to permit data replication. As 
has been mentioned before, there seems to be some will in the MLS 
community to move to better data standards. This is within the scope of 
RETS and has been for as long as I have been involved, from RETS 1.0 if 
not earlier.

Given the apparent willingness of the managers of the data to work 
towards better data, a natural place to handle that improvement is 
within the upcoming changes that are proposed for the RETS2 schema. 
While it is possible that this will prove to be reaching too far, typing 
and naming, it seems that this is an opportune time to make that 
stretch. We should at least make the attempt.

We seem to be getting mired in technical details rather than moving 
forward. At the absolute minimum, we need to be building some consensus 
on the element names. Toward that end, I would suggest that those with 
concerns about typing set that aside. Those with desire for greater 
typing to ignore that for the time being and both to focus on gathering 
the missing element names, clarifying element names and building the set 
of items that should be in the schema. Rather than arguing about the 
relative merits of pattern matching during the validation process, we 
should be debating the division between 
Properties/Listings/PublicRecords and similar discussions. What elements 
belong where? Do we have all the needed schema? Where is the 
element/attribute XXX? These and many other questions are where we need 
to focus our attention.

I don't deny that the process of typing may be difficult. I do believe 
that many things will fall out naturally. I do believe that there are 
potential solutions to resolve a large number of cases of mis-typed 
data. In the near term, however, let's get on with the job of 
identifying element names.

-- 
Paul Stusiak
Falcon Technologies Corp.