The Perception-of-Relationships Test (PORT) and Bricklin Perceptual Scales (BPS), 1961-2002

Current and New Empirical Data on 3,880 Cases

 

Barry Bricklin, Ph.D.

Gail Elliot, Ph.D.

 

Evidence-to-Conclusions Models: the Goal of Science

Knowing reliability and validity numbers, along with information about previously examined groups, cannot themselves tell a decision-maker whether a test is addressing the precise conceptual world in which there is interest.  What is required is the entire chain of reasoning that links evidence to conclusions.  This paper, which describes parts of an ongoing 7-year project, offers an evidence-to-conclusions chain for the Bricklin Perceptual Scales (BPS) and the Perception-of-Relationships Test (PORT) (Bricklin, 1984; 1989).

The PORT and BPS form the research-derived, data-based part of a comprehensive system: A Comprehensive Custody Evaluation Standard System, ACCESS (Bricklin & Elliot, 1995).  The tests measure the comfort and efficiency with which a child and his or her caretakers exchange a wide variety of interpersonal and emotional information within multiple family contexts.  They are not meant to assign parents to legal categories.  Such assignments are made by the court and depend upon information not obtainable from scientific models.

The following model will be used to organize the evidence-to-conclusions chain for the PORT and BPS.  It consists of four tiers.  The first tier consists of concepts, the second of principles.  Principles state the relations among concepts.  Empirical Equivalents, abbreviated EEs, define what one looks for in the world of sensory experience that exemplifies a concept.  What one chooses as the EEs to represent any concept cannot be determined by a scientific process.  All that can be claimed is that other choices may have created better or worse (i.e., different) results.  As Einstein put it, the ways we draw boundaries around our sensory worlds, the physical as well as the mental, are “free creations of the human mind” (Einstein, 1936; 1954; 1984).  Validation refers to the degree to which the relations among the EEs of the concepts correspond to the relations among the concepts as stated in the principles.

The predictor concepts of the tests aim to yield information on how effectively and comfortably a caretaker and child can exchange all manners of information (concerning the caretaker’s offering and modeling of competency skills, emotional support, consistency, etc.) given the child’s age, developmental status and idiosyncratic needs.  At a test level, the concept’s EEs are represented by the test items, a respondent’s responses to the items and the test’s scoring system.

Perhaps the most important tier for a decision-maker involves the EEs chosen to prove what a test claims to measure.  Here we refer to the precise way the independent validating-criteria are selected and measured.  It is with these data that the decision-maker can question whether the test is measuring the decision-area of concern.

Concepts and Empirical Equivalents (EEs) of the Tests

            The conceptual worlds of both the PORT and BPS aim to illuminate the degree to which a child can comfortably and efficiently exchange all manners of information with a parent (or any other assessed individual).  Such interchanges may involve the child’s seeking of psychological closeness and support, help with various tasks, or when a parent is teaching or modeling essential childcare behaviors (competency skills, trustworthiness, etc.).

One conceptual goal was to allow a child to respond nonverbally.  Nonverbal responses reduce loyalty conflicts.  The child is never asked to speak for or against a parent.  Nonverbal responses also help overcome limitations in a child’s expressive vocabulary.  Most (nonprogrammed) children apparently have, and use, a very limited number or words to describe their parents.  (Validity and clinical data given later further prove that our conceptual belief in the superiority of nonverbal over verbal data is well-placed.  They achieve much higher correlations with independently derived confirmatory evidence than do verbal data.)

The conceptual world seeks information from children so that it is possible to compare responses in which the child is consciously aware of what is being revealed to responses in which the child is not aware of what is being revealed, the aim being to help an evaluator detect which responses are driven not by a child’s actual interactions with a parent but rather by various parental pressures (e.g., bribery, intimidation, alienation, or by a child’s desire to save a parent seen as impaired).

            Test data were generated so that an evaluator could differentiate between responses likely to change situationally or rapidly from those of a more stable nature.  New test-retest data offered later, address this challenge.

            Conceptual EEs were selected that could assess the child’s interactions within various family subsystems (e.g., alone with a parent, with both parents simultaneously, within a total family grouping, etc.).  Some evaluators assert such information is not necessary because once the parents are divorced the child will not encounter these configurations.  Not only is this untrue, but ignores the fact that for years to come, perhaps forever, the child will deal with these configurations mentally.

The EEs selected allow a child a wide response range within assessed categories.  The child can express far more than “Mom is better than Dad.”  The child can say that in such-and-such an area e.g., helping with homework, Mom’s range of styles works to “this” degree, and Dad’s to “that” degree.

            The test EEs reflect a child’s developmental and idiosyncratic need-structure, so that an evaluator can ascertain what, for a given child, would be a useful trade-off.  For one child, a parent who has great organizational skills but is a bit cold emotionally may be a good trade-off, while this may not be true for another child, who may profit more from emotional support than parental organization skills.

The seven PORT tasks were chosen from more than 100 that were considered for inclusion in two pilot studies (described in the PORT Manual).  The ones retained were those able to pinpoint which parent engendered a specific test response in a child, and also achieved the highest (nonoverlapping) correlations with the validating criteria.  We also developed and tested a Family Systems Observation format in these pilot studies which will be described later.

            The BPS tasks were also selected because a child’s responses to them were clearly attributable to a specific parent.  The actual childcare areas were chosen in a way that represented the degree to which each conceptual target area appeared among legal statutes, case law and the writings of mental health experts i.e., the teaching, modeling, and offering of competency skills, admirable traits, and emotional support.

 

Validation Issues (Principles)

            In the custody field, everyone might agree on the criterion concept, something we call an arrangement in the “best interests of the child.”  If we now all agreed on exactly how we should measure this, that is, if we all agreed on what the EEs that represent the “best interests of a child” should be, all that would then matter is how accurately (and at what cost) our predictor EEs predicted them.  However, there is no such agreement (Krauss & Sales, 2000, pp. 845-862).  From the decision-maker’s perspective, no matter how accurate a prediction might be, if it does not represent the conceptual world in which there is concern, the predictive data are of no value.  The central question to a decision-maker is the degree to which he or she believes the EEs selected to represent criterion concepts faithfully represent the area of decision concern.

In the following paragraphs we will outline the major features we sought in our choice of criterion EEs as well as the sources of our criterion data.  We will later consider the degree of precision required of validating data in a child custody context.  First, and foremost, the focus was to be on selecting criterion EEs that reflect the comfort and effectiveness with which all manners of information are exchanged between a child and his or her caretakers.  This “goodness of fit” model reflects our conceptual premise that aside from exceedingly polarized extremes there is no meaningful way to formulate a list of  “admirable parental traits.”  Nor can such a list, if one existed, deal with the adequacy of cost-benefit ratios for a given child.

The second major factor was that the criterion EEs  be formulated from data gathered over lengthy periods of time (in our samples, 4 months to 7 years) and arise from multiple and independent sources of information (i.e., represent ecologic validity).

            The main sources of obtaining criterion data were as follows.  The original PORT study used a one-way mirror setup where trained psychologists observed children and their parents interacting in spontaneous, semi-structured and highly structured tasks. 

            We have always considered the richest source of validity data to arise from family therapy notes plus consultation with family therapists who had worked with children and their families over spans of not less than four months and up to seven years.  In these settings, the psychotherapists had opportunities to observe the children interacting with the parents in multiple subsystems (e.g., alone with a parent, with both parents together, with the entire family together etc.) and additionally heard significant numbers of facts obtained from other independent sources such as homes, schools, and communities.  Several psychologists have made the point that there are some criterion data that can only be obtained from psychotherapists, since they would be the only people who would be privy to the information (Miller, 1998).  Here are some examples of parent information that were shared with involved psychotherapists.

            “Whenever she asks me a question, I freeze up, retreat and get angry.”

            “I really don’t like it when he touches me.”

            “He reminds me of my husband, and that upsets me.”

            “I actually go to bed early so she won’t ask me to help her with her homework.”

It is unlikely that anyone except psychotherapists could assemble an array of data that could include such items.

            Another excellent source of criterion data would be custody evaluators.  Custody evaluators typically work with children and their families over many months, and obtain data from multiple and independent sources.  They typically have interview data, observation data, test data, and data obtained from multiple documents.  They also get to interview and observe many collateral informants.

Next would be mental health professionals who have had opportunities to collect data-rich information over significant spans of time.

            Last in quality would be courtroom judges.  However, please note that this is the criterion we are frequently trying to predict.  On the other hand, many of the courtroom judges who make designations have done so with exposure to the test data on which we sought validation.  Hence, agreement rates with courtroom judges are reported as “information.”

The EEs of the original PORT validity criterion concepts (Bricklin, 1989; Bricklin & Bricklin, 1999, p. 340) were a large number of (mostly non-verbal) behavioral interchanges between parents and children observed from behind one-way mirrors by trained psychologists.  This conceptual world was first articulated in the writings of Karen Horney.  Her system, though seemingly simplistic, posited that all interpersonal behaviors (inter- and ultimately intra-personal) can be characterized along three motor or movement dimensions.  These are: toward people (positive); against people (aggressive); away from people (fearful) (Horney, 1945).  Evaluator attention was to focus mainly on the child, since research interest was primarily on what impact parental behavior had on the child and not primarily on what parents knew or did.  Paul Ekman’s body of work, polygraph studies and the like, further point to the advantages of nonverbal over verbal data in assessing “gut-level” responses (Ekman, 1985; Ekman & O’Sullivan, 1991).

The Horney theory was operationalized in the following way.  The following positive categories apply to situations in which the child is either speaking (initiating or responding), just listening, or listening and acting:  smooth, regular breathing patterns; body movements fluid and non-hurried; relaxed and/or smiling facial muscles (no grimaces, etc.); leans toward other person; maintains reasonable eye contact (the evaluators were taught the difference between fearful eye-contact and relaxed eye-contact—the former is motivated by the child’s fear of saying or doing something the parent would object to); moves closer to or initiates physical contact; willing to be hugged; few signs of restlessness (even if a child is ADD-like, it may be that one parent has a far more calming influence than the other parent).

The following (positive) categories apply to situations in which the child is speaking, either initiating or responding: willingness to express annoyance, doubts or confusion (i.e., not trying to be the perfect little child); pauses without fear of losing caretaker’s attention; willing to ask questions; noting from whom the child most frequently and spontaneously seeks help.

The following (positive) categories apply to situations where the child is listening or listening and acting: accepts limits in mainly comfortable manner; muscularly comfortable with failures (no agitated moves); focused attention; facial expression animated and interested; no evidence of leaping-to-action before waiting for the entire “sent message” to be received; open and ready to receive information; willing to explore and take chances; willing to try novel approaches.  (In some of the original pilot-study research, separate accounts were kept of positive interchanges and negative interchanges.  It soon became clear that counting the number of positive interactions only yielded the same conclusionary information required as did counting both.)

The three observing raters watched as the parents and child interacted in spontaneous, semi-structured and structured situations.  Children were given easy and difficult tasks in the simultaneous presence of both parents, and recordings were made of how the parents were approached.  The tasks were selected so that they ranged, for a given child, from “easy-to-solve” to “impossible-to-solve.”  This allowed the observers to note not only how a parent exchanged information with a child when help was needed, but also when help was not needed. The observations were categorized as described below.  Interrater agreement was very high—in the 90 percent range—partly because the categories in which the data were summarized were formulated to match the requirements of the legal system (discussed later):  A > B (or B > A); A @ B; neither A nor B is adequate (Interrater agreement in the number of positive interactions noted were also good, in the 70 to 80 percent range.)  Differences were discussed and reconciled.  The observational rating scheme used in the PORT studies has remained largely intact until now, and has been the prototype of all future observational schemes.  These original criterion EEs achieved a 90 percent agreement rate with the PORT-selected parents-of-choice (POC), and the PORT has gone on to achieve similar rates of agreement with other criteria in about 1,508 cases.

For EEs used as validating criteria in subsequent research, parent-child interactions in a wide variety of sources were utilized.  The most important source was ongoing family therapy sessions, which lasted over time spans of two to seven years.  Since we wanted honest and spontaneous interactions, initial phases of data-gathering focused on non-adversarial families.  Other data were subsequently obtained from custody litigants.  Family therapy interactions obtained from non-warring families gathered over lengthy time periods yield information as close to real-life data as a mental health professional is likely to ever get, short of moving in with a family for several years.  Further, the samples represented a given child functioning within many different subsystems and under differing circumstances.

In the new data to be presented, each mental health professional who made a validity criterion designation had to have continuing contact either with the family of a tested child, or with one who had such contact.  Since we wanted ecologic validity, each validity designator was instructed to use all of the test, documentary, observation and other clinical/life-history information available (except for PORT or BPS scores).  This included numerous consultations with the mental health professionals who had continuing contact with each child and his or her family over the time-spans involved.  The exact questions asked of those who made criterion designations were as follows.  “Which parent (or other caretaker) overall offers and models competency skills, emotional support, discipline and structure in ways that are best suited to a child’s ability to use these responses effectively and comfortably?”  This was followed by a question asking whether any difference between the parents was narrow or substantial.  (“Narrow” was interpreted to mean the parents were approximately equal.)  We further asked whether both parents did not meet even minimal standards of “good-enough” parenting.

            This yields the following choices:  A is substantially better than B; B is substantially better than A; A and B are approximately equal; neither A nor B is even minimally adequate.  At no point did we, in this study, use the term “parent of choice” as was done in some of our earlier research, since the term was confused with that for a parent serving in some legally defined role.  Out tests measure the comfort and effectiveness with which a parent and child can exchange all manners of information.  Other criteria enter the picture when a decision-maker assigns a parent to a legal category.

Summarizing, the EEs of the conceptual targets of the independent validating criteria represent samples drawn from a wide variety of sources (i.e., pointing to convergent validity), are as representative of “real-life” as one is likely to get, were obtained over lengthy time periods and focused on parent-child relations within many family subsystems.

 

What Degree of Precision is Required of Validating Data?

            All mental health professionals who offer validity designations are directed to use judgmental categories that reflect the rather narrow range of choices utilized by our legal system.  Legal decision makers render decisions in two main categories: legal custody and a time-share plan (or physical custody).  In the former category, three choices are typically used: sole custody; shared custody; and very rarely, neither parent should have custody.

            As far as time-share plans are concerned, few judges venture beyond two choices: sole custody or joint custody.  If the former is chosen, the non-custodial parent is usually given a boilerplate arrangement in which he or she has the child every other weekend, plus an overnight in the middle of those weeks when he or she does not have the child for the weekend.  When complex time-share plans are awarded, they are almost always worked out by the parents and/or their attorneys, not by the data or the judge.  Judges rarely, on their own, venture beyond simplistic dispositional categories.  This constricted choice range has some advantages when transferred to a research setting, since it spares the participating professionals the need to make designations beyond those that are required e.g., it avoids hair-spitting arguments about how much better one parent is than the other.  In real life, mental health professionals can, and do, offer more complex and sophisticated suggestions.  Occasionally, judges heed them.

1.     Following, are data on previously tested groups, concurrent and future validity percentages and test-retest stability figures.  Brand new data address the challenge of identifying the conditions under which test data are likely to change over time (i.e., data that pinpoint unstable family systems) and when such changes should be viewed as true changes in the measured variables rather than errors of measurement.  Other new data address the degree to which the PORT is able to effectively assess how the child reacts to various family subsystems.

2.     PORT Normative Data (1961-1997), n=1,581

2-1.             Sex: 797 females; 784 males

2-2.            Age: Mean age 7.76; SD=0.17

2-3.            SES: Low-Middle to High-Middle

2-4.         Race: 98 percent Caucasian; 2 percent all other

3.     BPS Normative Data (1964-1997), n=2,389

3-1.            Sex: 1202 females; 1,187 males

3-2.            Age: Mean age 8.94; SD=2.40

3-3.            SES: Low-Middle to High-Middle

3-4.         Race: 98 percent Caucasian; 2 percent all others

4.     PORT Test-Retest Stability Data (1962-1997), n=21    (More data are given later.)

4-1.         If the Task-Difference-Score (TDS) (over a 6-month interval) was 0 or 1, there was a 10 percent chance the PORT Parent-of-Choice (POC) would shift.  If the TDS was 2 or more, the chance of POC shift was 2 percent.

5.      BPS Test-Retest Stability Data (1964-1997), n=20; n=33

5-1.         If the Item-Difference-Score (IDS) (over a 1-week interval, n=20 and a 6-month interval, n=33) was 0, 1 or 2, there was a 10 percent chance of POC shift.  If the IDS was 3 or more, there was a 2 percent chance of shift.

Please note again that the validity designations, presented next, were matched to the degree of refinement generally required in the legal system: parent A>B; parent B>A; Parent A @ Parent B; Neither A nor B is a good choice.  PORT and BPS data are summarized to yield a designation matching the first 3 of these requirements.  The system of which they are a part—ACCESS—addresses the last choice.  Those who made independent designations never did so on the bases of PORT or BPS data.

Please note, then, that the criterion designation categories yielded by the PORT and BPS do not yield a normal or symmetrical distribution.  Hence, percent-of-agreement is a reasonable way to express our results, and are inherently more user-friendly to many of those who would make decisions on their bases.  Further, it is a lot easier for non-statisticians to comprehend percent-of-agreement rates than product-moment correlations, where fairly large numbers frequently signify modest predictive accuracy.  A test-suggested Parent-of-Choice, abbreviated POC, is never, by itself or as part of our comprehensive system, used either to address legal ultimate issues or to assign a caretaker to a legal category.

6.     PORT Validity Data (1961-1997), n=1,381

6-1.         The percent-of-agreement rate is listed following the sample size.  Structured task problem-solving by children with access to both parents, observed from behind a one-way screen by two psychologists (1961), n=30, 90 percent; courtroom judges (1964-1981), based on all data available, n=45, 89 percent; agreement with BPS choices (1964-1981), n=23, 83 percent; courtroom judges (1981-1985), based on all data available, n=42, 95 percent; agreement with BPS choices (1981-1983), n=30, 84 percent; two psychologists, based on family therapy notes plus consultation with relevant therapists with families seen over two- to five-year intervals (1980-1985), n=30, 93 percent; courtroom judges (1986-1990), based on all data available, n=76, 93 percent; independent psychologists based on all clinical (except for PORT and BPS scores) and life-history data available (1995-1997), n=1,038, 89 percent.

7.      BPS Validity Data (1964-1997), n=2,279

7-1.         Agreement with PORT choices (1964-1981), n=23, 83 percent; two psychologists, based on family therapy notes plus consultation with relevant therapists with families seen over two- to seven-year intervals (1980-1983), n=21, 100 percent; courtroom judges (1980-1983), n=30, 90 percent; “Would” questionnaire choices (a “disguised” semi-projective test, asking what Mommy/Daddy would do in certain situations e.g., “You get a bad mark on a test”) (1980-1983), n=23, 87 percent; PORT choices (1981-1983), n=30, 84 percent; courtroom judges based on all available information (1984-1990), n=179, 96 percent; independent psychologists based on all clinical and life-history data available (1988), n=141, 97 percent; independent psychologists based on all clinical and life-history data available (1992-1995), n=1,765, 88 percent; independent psychologists based on all clinical and life-history data available (1995-1997), n=67, 87 percent.

8.      PORT Normative Data (1997-2002), n=127

8-1.         Sex: 61 females; 66 males

8-2.         Age: Mean age 7.87; SD=2.101

8-3.         SES: Low-Middle to Upper-Middle

8-4.         Race: 92 percent Caucasian; 8 percent all other

 

9.     BPS Normative Data (1997-2002), n=93

9-1.         Sex: 47 females; 46 males

9-2.         Age: Mean age 7.88; SD=1.473

9-3.         SES: Low-Middle to Upper-Middle

9-4.         Race: 92 percent Caucasian; 8 percent all other

 

10.  PORT Test-Retest Stability (1997-2002), n=127

10-1        If the TDS is 0 or 1, there is a 21 percent chance of POC shift over an 8-month interval.  If the TDS is 2 or more, there is a 3 percent chance of POC shift.

 

TABLE 1

PORT Task-Difference-Scores (TDS) and Changes in Parent-of-Choice (POC)

from Test 1 to Test 2

 

11.  BPS Test-Retest Stability (1997-2002), n=93

11-1.      If the IDS is 0, 1, 2 or 3, there is a 19 percent chance of POC shift over an 8-month interval.  If the IDS is 4 or more, there is a 3 percent chance of POC shift.

 

 

 

 

 

 

 

 

 

 

 

 

 

TABLE 2

BPS Item-Difference-Scores (IDS) and Changes in Parent-of-Choice (POC)

From Test 1 to Test 2

IDS	Frequency             at Test 1	Frequency             at Test 2	POC   Changed	Percent                POC Changed	Percent                                    POC Unchanged                        with  > IDS
32-25	6	0	0	0.00	100.00
24-16	9	1	1	11.10	93.30
15-4	57	71	1	1.80	97.20
3	3	4	1	33.30	96.00
2	4	3	1	25.00	94.90
1	6	7	1	16.70	94.10
0	8	7