Dr. Neil Clark Warren, eHarmony founder
Last Tuesday I posted Part 1 of this 2 part series. Here is the remainder of my study into online dating and the scientific validity of eHarmony’s 29 dimensions of compatibility.
To pick up where we left off, my investigation into the merits of online dating (including but not limited to my personal experience) led me to conclude that it isn’t an ill advised strategy to meet people. Many who have used it (and especially those who were willing and able to pay the monthly fees for the better online dating services) reported at the very least that the efforts resulted in a date. Meeting people online can be at least equally effective to the strategies employed by single people before the internet era. If anything, I might argue (although I lack tangible data to support) that online dating opens up possibilities for connections to some groups previously disenfranchised by the stereotypical dating scene of past decades, such as the very shy and/or the more socially uncomfortable.
However, some online dating sites, like eHarmony, make claims that go beyond this. eHarmony claims that they have developed an algorithm that will match singles based on “29 dimensions of compatibility”. This algorithm is ‘scientifically’ designed based on the experience and research of the site’s founder, Dr. Neil Clark Warren. Dr. Warren (pictured above) told USA today back in 2005,
There’s a way in which, a little paternalistically, we say we have discovered, on the basis of our research, what is required to make a marriage great. And we’re going to help with that. Our way….We do try to give people what they need, rather than just what they want”
The instrument is a 258 item questionnaire said to measure the 29 dimensions of compatibility, which are divided into several “core traits” and “vital attributes”. Their theory is that most relationships start out as chemistry; there site offers a match in compatibility. The core traits involved include temperament, adaptability, curiosity, intellect, values and beliefs, relationship skills and conflict resolution. When you complete the questionnaire, you’re offered a personality profile, and other singles are matched based on their system.
It was my goal to test eHarmony’s claim that their scientific algorithm would find better matches than other sites where I would conduct my own searches. My method seemed simple enough. I wanted to take the personality questionnaire, noting the sort of questions it asked of me, and get an estimated figure of the number of search results that turned up within New York City. I then planned on returning to some of my old online dating accounts, setting the usual rigid search filters, and see how many search results turn up in New York City. In addition, I wanted to look at the top ten search results for all pages and see if I could objectively estimate how interested I would be in each partner.
My hypotheses were pretty straightforward. First, I estimated that the search results produced by eHarmony would not be significantly less in volume than the other dating sites that allow me to do the searching directly. I would conclude from this that if their method was presumable better, then I should see fewer overall search results. The second part, that the 29 dimensions of compatibility would enable more precise results, meaning that I expected to find a greater proportion of men I might be interested in dating (if of course I wasn’t already hopelessly in love) amongst the top sample of results.
I’m sorry to report that I was unable to conduct my analysis. After having spent nearly an hour completing the eHarmony questionnaire; I was rejected by eHarmony.
I’m apparently not alone, eHarmony estimates that they reject 20% of candidates who complete their questionnaire.Their rejection letter writes, “We are so convinced of the importance of creating compatible matches to help people establish happy, lasting relationships that we sometimes choose not to provide service rather than risk an uncertain match.”
Reasons for rejection are varied. A general list of such reasons include testing low in self-concept, emotional status (happiness), character (honesty and trustworthiness), obstreperousness (don’t fret, I linked it for you), emotional management, conflict resolution or if they detect an unhappy childhood or a conflictual relationship with your parents. The argument is, if eHarmony doesn’t think you’re ready for marriage, they won’t pair one of their high paying customers with you.
And marriage is the key with eHarmony. According to their website , 2% of marriages in the United states are the result of an eHarmony match.
Don’t be misled, this isn’t a vendetta against eHarmony for rejecting me. Bear in mind first that I’m very happy in my relationship. I went to their site strictly because I suspected something was awry with their claims and I wanted to look into matters for myself. The rejection was actually serendipitous, as it led me to research further their instrument and assessment criteria.
For starters, as I completed their questionnaire, I took note of the items. Most of the instrument is pretty typical of psychological personality testing. For starters, participants rate how well words like ‘frugal’, ‘intelligent’, ‘attractive’, or ‘loyal’ describe them. Participants were also asked to imagine what words their friends would use to describe them. Several likert-type statements, such as, “I am satisfied with my level of emotional development”, “I tend to think “outside the box”", “I like to look at people of the opposite sex” are offered and participants indicate the degree to which they agree with the statement. The questionnaire then goes on, and on. After a certain point I stopped taking note of the items (and what my answers were, which might have led to my downfall).
Here’s the thing. Whether this instrument tests what it actually claims to test has not been established. If I propose a questionnaire or survey is measuring depressiveness in an individual, it should be able to demonstrate that it does with measures of validity and reliability (see Which Character from Seinfeld are you?). Epstein (2007) also wisely points out that if such tests had been conducted to measure the veracity of the instrument, the results should be published, and no such publications exist.
But forget that the instrument matches singles for a moment and reflect solely on the individuals this website has rejected. If eHarmony claims that a person scored low in ‘character’ and as such could not be offered their matching services, it is not unreasonable to expect eHarmony to demonstrate that their instrument is accurately detecting poor character. To not do so isn’t just practicing poor science, it’s practicing no science at all.
However, given the procedures observed when the eHarmony labs do sit down and conduct a study, I don’t blame them for avoiding thorough science altogether. For starters, in 2004 eHarmony researchers presented at the annual convention of the American Psychological Society a study conducted by their researchers. The conclusion of the paper was that couples who married after being matched on eHarmony were significantly happier in their marriage than couples who met through other means. A review of their methods, however, and some glaring problems are clear. To begin, the average length of the marriage for the eHarmony sample (out of a year) was .5, while it was 2.12 for non-eHarmony matched married couples. This difference, presented in their own paper, was significant (P<.001). The two samples also differed significantly in mean age at wedding (eHarmony= 40.4, others= 33.00, P<.001), years dating prior to marriage (eHarmony =.23, others =3.15, P<.001), level of education completed (for example, eHarmony =6% had less than H.S. others =22%; eHarmony =20% held a Masters degree, others =10%, p<.001) and income (for example, eHarmony =34% earning between $60,-125,000, others =22%, P=.015).
Their ANCOVA results show the numbers of years dating prior to marriage and the length of marriage were all significant covariates of the significant mean difference. For those unfamiliar with the procedure, ANCOVA is an analysis of covariance measuring the degree to which a variable contributes to the variation in a dependent factor. Covariates are other variables incorporated into the model that can either be eliminated (through non-significance) as other potential factors contributing to the outcome of the results, or, as in these results here, demonstrate they too contribute to the variation of the observed outcome.
In short, how long couples dated before they married and the length of the marriage at the time they completed the questionnaire are also factors capable of explaining the difference between the two samples. Given the incredibly divergent differences between the samples, the fact that eHarmony couples were more likely to be marrying for the second time (mean number of previous marriages, eHarmony =.5, others =.29, p<.000), at an older age, dated for far less time, and were far earlier in their respective marriages, it isn’t really all that surprising that they would still be in a place where they feel more emotionally happy and affectionate in their relationship.
So, to sound even less technical, duh.
So, with respect to eHarmony’s claim, that their method is a scientific one capable of matching singles to a person who is superior in compatibility, could not be tested for this investigation. However, it is unlikely that they are able to given the instrument they employ to measure and characterize service applicants has not been tested to determine that it actually measures 29 dimensions of anything. If the instrument is ambiguous, whatever conclusions are drawn from the instrument are equally ambiguous. In addition, if eHarmony wishes to present itself as a company capable of producing a ‘happier’ marriage, their data do not support this claim. The comparison sample differed far too much from the eHarmony sample to allow for internal validity of the study. For an investigation to possess internal validity, there must be no other potential explanatory factors of the outcome. Given that their own statistical procedures demonstrated the significant co variation of several other explanatory characteristics, they lack this important criteria of properly rigorous science.
In some form of summary and conclusion, dating and relationships might be worth studying from a scientific perspective. And perhaps there is some way of measuring the likelihood that two people are right for each other. But, if such studies occur and such findings exist, it is up to scientists to determine the quality of the procedures and data not online dating moguls.
References
Epstein, R. (2007) The Truth about Online Dating Scientific American Mind, feb/march, 2007
Miller, E. (2009) eHarmony Is Just Not That Into Me, Politics Daily, 9/09/09
Kornblum, J. (2005) eHarmony: Heart and Soul, USA Today, 5/28/2005