 |  | SIGNAL POSTED BY carl | 08.29.02 @ 04:14 PM
My Fellow .Organisms
[ FILED UNDER: Notices » org » ] |
 My Fellow .Organisms:
Many of you have sent signals asking what we thought of the
ICANN Preliminary Evaluation. Three committees
([1],
[2],
[3])
generated rating matrices which was summarized in a
staff-generated meta-matrix
([4])
which yielded a single metric.
While the IMS/ISC bid received
extremely high marks in supporting areas such as vision, experience,
innovation, service, and commitment, the technical evaluation rated us
somewhere
between totally clueless and moderately brain dead.
Rather than engage in a blow-by-blow
evaluation of the evaluations, it seems more appropriate to pull up to 50,000 feet
and explain why there is a fundamental difference of opinion about how
to do what we call in the trade "technical due diligence."
Since "a demonstrated ability to operate a registry of this
scale" is the primary technical criterion, all the supplicants
prepared appropriate paper trails to document such a capability.
Much to our puzzlement, we haven't received a single email,
phone call, fax, or chat room request to view source code,
do a site visit, examine log files, or get an ssh login to
look around our systems.
While a total lack of technical due diligence is not
unheard of in such procurements, it was also a
surprise to see the paper trail taken at face value in two other areas:
 1. When reading a report, it's always nice to know a bit about
the authors. We're not really familiar with the Gartner Group or
the MIS managers who prepared the technical evaluations, so we
prepared a little
"getting to know you"
presentation.
While we've never been a Gartner client, we were surprised that Gartner
did not disclose that it has had significant business relationships with
NeuStar, VeriSign, and Register.Com.
2. "A demonstrated ability" was shown by many supplicants based
on prior experience in the business. We looked around, but
it appears that
none of the reports that document
actual
performance results
of the established players are on-line.
In short, it appears there's a bit of a traffic jam on the paper trail,
but luckily there are several routes to our destination.
After
sitting in traffic for the last few months inhaling the exhaust from
the money-guzzling vehicles driven by .commies and lawyers, we've decided
to switch metaphors as a way of helping to save the environment.
Outside the . beltway, fancy cars and loud meetings just don't have the
same appeal. We look instead for our inspiration to the family farm.
T.S. Eliot once
asked, "what are the roots that clutch,
what branches grow?" [audio] ![[audio]](http://town.hall.org/images/speaker.gif)
- Our roots are our current services operating on the Internet. You are
invited to participate in our registry interoperability
testbed,
comment
on
technical notes,
and examine our
operational statistics.
- Our branches are
the other people who use our code. You are invited
to download our software for DNS
and DHCP, and we are pleased
to announce that training on our OpenReg open registration system
will start early next year for our colleagues who run ccTLD and gTLD
registries, as well as those that operate registries in corporations,
universities, or other organizations that need to systematically delegate and
allocate
names.
In any case, the popular
vote is in and all we can do is wait for the Electoral College to decide the future of the
.org TLD.
Thanks for your support.
| Comments |
|
 |
Subject: Gartner..
Posted by Roy Pennington at 09.15.02 @ 04.03 PM |
Why people trust a analyst-for-hire company that can benefit from the outcome, is beyond me.
Especially Gartner, who have been known to be, occasionally, far from objective.
The fact Gartner has business dealings with the some of the bidding groups involved, should raise red flags.
|
Subject: The real measure
Posted by Chris Petrilli at 09.03.02 @ 08.08 AM |
The real measure of success has ceased to be technical, but is now monetary. One can not imagine the logical pretzel one has to tie ones mind into to come up with Carl and Paul as anything less than A+++ stars technically. To argue anyone else is "better" is to commit the most gross demonstration of a finger on the scale of measurement.
Then again, did anyone expect ICANN to be fair?
|
Subject: "Traditional weight and scores" -- hardly an unbiasable methodology...
Posted by Jerry Asher at 08.31.02 @ 11.26 AM |
I have used and performed evaluations using the "traditional weight and scores" approach. The approach per se' is hardly above bias. In fact in the hands of skilled operators, it can easily be used to provide a preselected, biased answer and cover that answer with a thick veneer of credibility in the form of a brand name consulting report. In the case of a report produced by a brand name consulting company with undisclosed financial associations with some of the evaluees, I would think the report would be highly suspicious on its face. Has Enron/Anderson/WorldCom/Quest/Adelphia/... taught ICANN nothing?
I would suggest the team enumerating the criteria to be judged be different from the team measuring the results. The team assigning weights should assign weights openly, perhaps as a poll of known experts (in this case, CTOs, CIOs, Professional Engineers, ...?)
Traditional weight and scores can be used to provide an accurate, credible, repeatible result, but if so, I would suggest using techniques that any Ph.D. student, involved in experimental and statistical methods would be familiar with.
One, are the measures credible and reliable? Are the factors even measurable? The measures should not be the result of two allegedly unbiased experts, but the measures should be based on trainable criteria that can be taught to random, intelligent observers. Many of these observers can be used to measure each factor, and this results in being able to evaluate the measurability of each factor itself in terms of interrater reliability. How close to random, unbiased observers measure the factor. If a factor can be measured this interrater reliability should be high.
Two, double blind testing and scoring. Each answers in the proposals should be broken apart, formatted in a neutral fashion, and handed in a random order to a scorer. It should not be possible for scorers to determine who wrote the answer.
"Traditional weight and scores" is useful as a guideline, but is hardly an independent measuring system.
In this case, I am hardly surprised that a brand name consulting company, picks on of its own. Has such a method ever picked an outsider?
|
Building Public Parks for the Global Village Since 1993. 