Stephen Vasciannie ANGELA PATTERSON, the General Manager CVM Television Limited, has responded to my column on 'The Media Ratings War', published in last Monday's Gleaner. Her response encourages further debate about important questions for the media, including especially the use of statistics in advertising.
Although I am responding directly to Mrs. Patterson's letter in last Friday's Gleaner, I hope this column will not be read as an issue concerning personalities. My overall impression from the outside (and primarily as a regular television viewer) is that Mrs. Patterson's stewardship at CVM TV has been good. Consider, for example, the current netball coverage, the innovative approach to sports reporting with 'Where are they now?', 'On Court', 'It all begins here', and the pronounced efforts to reveal community developments both in sports and in public affairs.
So then, the issue at hand is not Mrs. Patterson's general performance, nor is it whether CVM TV is fulfilling its mandate. Rather, it is a more narrow, but nonetheless important, point: CVM TV continues publicly to make claims about its viewership which do not stand up to even the most elementary standards of statistical analysis. This is cause for disappointment -- media houses need to maintain credibility, and superficial claims about popularity do nothing to promote such credibility.
ERROR NUMBER ONE
The CVM advertisements are based on a survey done by the JobBank. Here, with clarifications, are the main problems I have concerning the JobBank survey:
The JobBank findings are based on a very skewed sample of television viewers. As indicated by CVM itself, the JobBank survey was conducted among persons in two Jamaican cities and two large towns. That alone tells us that the sample is not randomly drawn -- Clear Error Number 1 from a statistical viewpoint.
By confining itself to two cities and two large towns, there will be obvious statistical bias. Viewing patterns for urban as against rural communities may well be different. For example, as between urban and rural viewers, the former are more likely to have access to cable television, and this could influence viewing patterns concerning CVM and TVJ: to illustrate, some urban dwellers may well say why watch Judging Amy or Boston Public on local television when I can catch more recent episodes on cable?
Moreover, city and large town coverage may not present the full viewership situation for technical reasons. One station may have greater penetration into deep rural areas because it has stronger or more strategically placed transmitters throughout Jamaica. This kind of point is largely ignored by the JobBank's approach: some deep rural dwellers certainly visit cities and large towns, but in probability terms, they are less likely to be picked up in the survey than persons who actually live in the two cities and two large towns.
With respect, Mrs. Patterson's response to this set of considerations -- namely, that the JobBank was not doing a demographic survey, and that no survey ever goes to every town or city -- misses the point. The deliberate selection of four of the largest cities and towns means that the JobBank survey is essentially a survey of urban perspectives, and not a survey of all island viewership patterns.
There are serious problems of internal consistency in the JobBank survey. Different sample sizes are used for different questions in the survey. For one question, up to 370 persons were interviewed, for another 355, for a third 253, a fourth 202, a fifth 198, and so on. Clear Error Number 2.
SAMPLING
In statistical sampling, there is, by definition, a relationship between the sample size and the overall population being studied; and different sample sizes predict different degrees of accuracy for the study. To ignore this fact, by having different sample sizes just so, is to promote guesswork, not statistical certainty. Mrs. Patterson's response does not address this problem.
The results obtained by the JobBank for the overall 'viewership' of CVM versus TVJ do not appear to coincide with the results obtained by the JobBank for 'viewership' of the main programmes surveyed on an individual basis. Likely error number 1, but silence from CVM.
In respect of sports programmes between 7 p.m. and 9 p.m., the JobBank survey indicates that everybody in Jamaica watches either CVM Sports Watch, TVJ's Prime Time Sports, or both programmes. This, I dare say, must be wrong: Clear Error Number 3. And, by the way, the JobBank survey also indicates that nobody watches both CVM's Talking Sports and TVJ's Eye on Sport regularly: Likely Error Number 2. Silence from CVM.
The main point used in CVM TV's advertisements about the JobBank survey is that CVM TV leads TVJ in six of seven viewership categories based on different age groups. This is an odd classification for CVM to adopt because, in all likelihood, people from all age groups watch both CVM and TVJ at different times. What advertisers will benefit from most, therefore, are popularity ratings in specific time slots.
Alas, the JobBank survey simply asks the broad question: "Which is your favourite local TV station?", and uses this to conclude that CVM TV leads TVJ in six viewership categories. How can this be so? This is a case of asking one question, and using the response to answer a different question. I do not always watch my 'favourite' station, and indeed, I may not have a favourite station, but rather 'favourite programmes', so a question about my favourite station actually says rather little about my viewing pattern. Clear Error Number 4. This point was not fully developed in last week's column.
Generally, Mrs. Patterson advances the criticism that I do not have all the information contained in the full body of the JobBank Report (she says that I have admitted this). She reaches this conclusion because I indicated in my penultimate paragraph that "it would be a good idea for the JobBank to provide a full report on the methodology used". Mrs. Patterson's position does not really follow; as it is a non sequitur, I will not dwell on it.
CLEAR ERRORS
Rather, I would emphasise that EVEN WITHOUT THE FULL REPORT, one can identify several clear methodological errors in the JobBank approach. Last week I put forward quite a few of these errors, and this week I have reiterated them. Mrs. Patterson cannot reasonably avoid addressing these errors by arguing that I do not have all the relevant information, or by implying that because some surveys are imperfect, this may somehow justify the errors in the CVM/JobBank approach. The points made above are not undermined by any of the elements in Mrs. Patterson's response.
Finally, Mrs. Patterson refers to my point about objectivity. My view is that if (a) CVM commissions a survey, and (b) this survey comes back with results complimentary to CVM, and (c) this survey goes against another survey (for a fairly recent earlier period), and (d) this survey is constructed on the basis of clear statistical errors and generates implausible results, I would certainly regard it as suspect.
Everybody watches sports news, but nobody watches both sports analysis programmes regularly?! Res ipsa loquitur -- the facts speak for themselves.
Stephen Vasciannie is Professor of International Law at the University of the West indies and Consultant in the attorney General's chanmers
