Journal of Epidemiology and Global Health

Volume 4, Issue 1, March 2014, Pages 29 - 34

Measuring the accuracy of a point system to diagnose tuberculosis in children with a negative smear or with no smear or culture

Authors
Constantino Giovani Braga Cartaxoa, constancartaxo@gmail.com, Laura C. Rodriguesb, laura.rodrigues@lshtm.ac.uk, Carolina Pinheiro Bragaa, carolinapbraga@gmail.com, Ricardo Arraes de Alencar Ximenesc, *, ricardo.ximenes@pq.cnpq.br
aDepartamento Materno-Infantil, Universidade Federal da Paraiba, Brazil
bFaculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, United Kingdom
cDepartamento de Medicina Tropical, Universidade Federal de Pernambuco, Brazil
*Corresponding author.
Corresponding Author
Ricardo Arraes de Alencar Ximenesricardo.ximenes@pq.cnpq.br
Received 11 December 2012, Revised 30 September 2013, Accepted 7 October 2013, Available Online 15 November 2013.
DOI
10.1016/j.jegh.2013.10.002How to use a DOI?
Keywords
Tuberculosis; Childhood; Scoring system; Validation study
Abstract

In Brazil, a scoring system was adopted to diagnose tuberculosis in childhood. This study determined the accuracy in diagnosing tuberculosis in children with either a negative smear or with no smear or culture conducted in a reference center in João Pessoa Paraíba – Brazil. It is a phase III validation study, using a cross-section design. The study population consisted of 167 patients attending the outpatient clinics suspected of having tuberculosis. The reference standard for the diagnosis of tuberculosis was a blind and independent review of the medical records, radiology and tuberculin test by two experts. Of the 167 patients, 60 were considered to have tuberculosis (by the reference standard diagnostics). The results for the scoring system with the cut-off of 30 points were: sensitivity 78.57% (95%-CI: 65.56–88.41%), specificity 69.16% (95%-CI: 59.50–77.73%), positive predictive value (PPV): 57.14% (95%-CI: 45.35–68.37%), negative predictive value (NPV): 86.05% (95%-CI: 76.89–92.58%), likelihood ratio (+): 2,55, pre-test probability: 34.36%, and post-test probability (+): 57.14%. This supports the current recommendation for the use of this scoring system in Brazil and similar sites with the cut-off of 30 points. However, as the discriminatory power of the point scoring system may vary across settings, it would be advisable to replicate this phase III study in different settings.

Copyright
© 2013 Ministry of Health, Saudi Arabia. Published by Elsevier Ltd.
Open Access
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction

The diagnosis of tuberculosis in children remains a challenge. Symptoms are non-specific and, in the early phases of the disease, up to 50% of the children may be asymptomatic [1,2]. In spite of the advances in tuberculosis diagnosing, no test is highly accurate in children [3]. The presence of acid-fast bacilli on sputum smear microscopy–the standard diagnostic criteria for adults–is rare in childhood. Recently, there has been an increasing interest in defining diagnostic criteria for patients with negative smear and culture [4]. In Brazil, a scoring system was adopted to diagnose tuberculosis in childhood in 2002 [5]. In childhood, bacteriology is positive in only 5% of cases and culture in 25% [6]. Diagnosis in children therefore must be based on clinical and radiological findings, tuberculin test results, epidemiological links, and response to treatment using algorithms or personal experience [6,7].

Diagnostic scoring systems for the diagnosis of tuberculosis were developed, but none are used routinely. All scores have limitations [810]: low sensitivity [11,12]; inclusion of smear or culture or histopathology as a criteria [10,11,13], or response to antibiotic therapy or anti-tuberculosis treatment as a step to diagnosis, and postponing the diagnosis of tuberculosis [10,14]. The scoring system adopted in Brazil in 2002 to diagnose childhood tuberculosis in children and adolescents aged 15 years or less has rarely been evaluated [1518] and was tested retrospectively, mainly with hospitalized patients, which has clear limitations [9,15,17]. The study reported in this paper aims to identify the accuracy of the scoring system proposed by the Ministry of Health to diagnosis tuberculosis in children aged 15 years or less with negative smear and culture attending outpatient clinics.

2. Methods

The study was conducted during the period 2003–2005, in a Reference Center for infectious diseases in João Pessoa Paraíba – Brazil. It is a phase III validation study, according to the Sackett and Haynes (2002) classification [19], using a cross-section design. In phase III studies, all individuals enrolled are suspected of having the disease and the investigation is carried out to verify if the test (the scoring system) distinguishes those with and without the target disorder. It was a delayed-type cross-sectional study in which the follow-up procedure was aimed at retrospectively assessing the health status at time zero, as a substitute for establishing the reference standard diagnosis of tuberculosis immediately at time zero itself [20].

Children and adolescents were considered suspects of tuberculosis according to the guidelines of the Ministry of Health [5]. In short, tuberculosis suspects were those children and adolescents who had been symptomatic for more than 15 days or who had contact with a respiratory symptomatic patient.

The reference standard for the diagnosis of tuberculosis was a blind and independent review of the medical records, radiology and tuberculin test by two experts in childhood tuberculosis disease. There was an independent assessment of the scoring and the reference standard. The experts were blind with regard to the result of the scoring for each patient, and the interviewer responsible for the application of the score was blind to the diagnosis established by the experts. Experts and the investigator responsible for the application of the score were not informed as to the diagnosis given by the attendant physician or if patients had started treatment for tuberculosis. Disagreement between the two experts was solved by a third expert who would review all discordant diagnoses. Two experts were specialists in pulmonology (by the Brazilian Society of Pulmonology and Tisiology), and one in pediatric infectious diseases (by the Brazilian Pediatric Society) and had experience in attending patients at the hospital and at the outpatient pediatric infectious disease and pediatric pulmonology clinics. Children were classified as positive by the reference standard if, after a period of six months, they had the diagnosis of tuberculosis by two (of three) experts. The steps for selection and follow-up of patients are summarized in Fig. 1.

Fig. 1

Flow diagram of the steps for selection and follow-up of patients.

Smear microscopy for acid-fast bacilli and culture for Mycobacterium tuberculosis was requested of all patients. A total of 68 patients were able to expectorate voluntarily and had the smear microscopy and culture performed, four of them had a positive smear and none had a positive culture. The four patients with a positive smear were excluded from the study.

The sample included 163 patients aged less than 15 years attending the outpatient clinics suspected of having tuberculosis disease. A total of 56 cases were diagnosed as tuberculosis disease according to the reference standard. The data were collected in a standard questionnaire applied by the investigator to the patient or caregiver. Sensitivity, specificity, predictive values, likelihood ratios and pre- and post-test probability were calculated for two score cut-off points: 30 and 40 points. All children in the study were followed up for 6 months, with complete treatment for those diagnosed with TB disease by the attending physician. None of the patients included in the study had, nor developed during the follow-up period, any sign or symptom of extra-pulmonary tuberculosis; lymph node, pleural, or other organs or systems involvement were not diagnosed. No child died. The scoring system is demonstrated in Table 1.

Score
Clinical manifestations
  Fever or cough, lost energy, expectoration, weight loss, night sweats >2 weeks +15
  No symptoms or symptoms <2 weeks 0
  Respiratory infection improving with or without antibiotic therapy for common bacteria −15
Thoracic X-ray
  Enlarged hilum or miliary pattern +15
  Exudate or patch shadow (with or without cavitation) unaltered >2 weeks or worse with antibiotic therapy for common bacteria +15
  Exudate or patch shadow (with or without cavitation) <2 weeks +5
  Normal −5
Contact with tuberculous adult
  Close, <2 years +10
  None or occasional 0
BCG vaccination and tuberculin test
  BCG ⩾ 2 years or no BCG (⩾10 mm) +15
  BCG < 2 years (⩾15 mm) +15
  BCG yes or no (5–9 mm) +5
  BCG yes or no (⩽5 mm) 0
Nutritional status
  Severe malnutrition (grade III) +5
  Eutrophic or no severe malnutrition 0

Score interpretation: ⩾40: PTB very likely; 30–35: possible PTB; ⩽25: PTB unlikely.

Table 1

Score system adopted in Brazil to diagnosis tuberculosis in childhood in 2002. Ministry of Health, Brazil [4].

This project was approved by the Ethics Committee of the Hospital Universitario Lauro Wanderley Universidade Federal da Paraiba (registration CEP/HULW 371/2004).

3. Results

The study was conducted on 163 patients suspected of having tuberculosis, of which 60 were considered to have tuberculosis according to the expert review. The average and median age of the children and adolescents were, respectively, 88.69 (SD 46.28) and 91.1 months, with a minimum age of 0.3 and a maximum of 189 months. Of the total of 163 children, 7 (4.29%) were less than 12 months in age, 43 (26.38%) were aged 12–59 months, 66 (40.49%) were aged 60–119 months, and 47 (28.83) were aged 120 months or more.

For 30 as the cut-off (Table 2), the sensitivity was 78.57% (95%-CI: 65.56–88.41%), specificity 69.16% (95%-CI: 59.50–77.73%), positive predictive value (PPV): 57.14% (95%-CI: 45.35–68.37%), negative predictive value (NPV): 86.05% (95%-CI: 76.89–92.58%), likelihood ratio (+): 2,55, pre-test probability: 34.36% and post-test probability (+): 57.14%. Using 40 points as the cut-off (Table 3), the values were sensitivity: 48.21% (95%-CI: 34.66–61.97%), specificity: 87.85% (95%-CI: 80.12–93.37%), PPV: 67.50% (95%-CI: 50.87–81.43%), NPV: 76.42% (95%-CI: 67.93–83.61), LR (+): 3.87, pre-test probability: 34.36% and post-test probability (+):76.42%.

Cut-off Reference standard Total

Case Control
>30 points 44 33 81
<30 points 12 74 86
Total 56 107 163
FONTE: Hospital Clementino Fraga João Pessoa-PB, January 2003/October 2005.
Sensitivity 78.57% 95%-CI: 65.56–88.41%
Specificity 69.16% 95%-CI: 59.50–77.73%
Positive predictive value 57.14% 95%-CI: 45.35–68.37%
Negative predictive value 86.05% 95%-CI: 76.89–92.58%
Likelihood ratio (+) 2.55 95%-CI: 1.86–3.49
Likelihood ratio (−) 0.31 95%-CI: 0.18–0.52
Probabilities
Pre-test (+) 34.36% Post-test (+) 57.14%
Pre-test (−) 65.64% Post-test (−) 86.05%
Table 2

Analysis of the accuracy of a point system to diagnose tuberculosis in children with negative smear and culture with cut-off of 30 points.

Cut-off Reference standard Total

Case Control
>40 points 27 13 44
<40 points 29 94 123
Total 56 107 163
FONTE: Hospital Clementino Fraga João Pessoa-PB, January 2003 a October 2005.
Sensitivity 48.21% 95%-CI: 34.66–61.97%
Specificity 87.85% 95%-CI: 80.12–93.37%
Positive predictive value 67.50% 95%-CI: 50.87–81.43%
Negative predictive value 76.42% 95%-CI: 67.93–83.61%
Likelihood ratio (+) 3.87 95%-CI: 2.23–7.07
Likelihood ratio (−) 0.59 95%-CI: 0.45–0.77
Probabilities
Pre-test (+) 34.36% Post-test (+) 67.50%
Pre-test (−) 65.64% Post-test (−) 76.42%
Table 3

Analysis of the accuracy of a point system to diagnose tuberculosis in children with negative smear and culture with cut-off of 40 points.

The concordance between the two experts was low (observed agreement: 61.7%, kappa of 15%). A third expert reviewed just the discordant diagnoses (n = 49). In the analysis of this subgroup, the observed agreement between the third expert and expert #1 was of 41.3% and between the third expert and expert #2 was of 74.5%.

4. Discussion

Sensitivity of 78% and specificity of 69% are low, but in the acceptable range, for a cut-off point of 30. The low expert’s concordance (observed agreement: 61.7%, kappa of 15%) may constitute a limitation of this study, although it does highlight how difficult it is to diagnosis tuberculosis in childhood. Disagreement between the two experts was solved by a third expert who reviewed all discordant diagnoses.

Some characteristics of the patients may be behind the agreement rate observed, such as the occurrence of patients with mild symptoms, with symptoms lasting less than 15 days, with normal chest radiograph, or with the presence of non-specific lesions without cavitation. Approximately 80% of the patients had a history of previous contact with a patient with tuberculosis.

Outpatient clinic studies using experts’ reviews as a reference standard are rare. Tuberculosis validation studies are, generally, hospital-based, with bacteriology and histopathology as the gold standard. This would obviously not be appropriate in this case where there is a search for a solution to the diagnosis of bacteriologically negative cases [11]. According to Sackett’s classification, this would result in limited external validity of the findings because of selection and spectrum bias [19].

There are a few studies discussing the accuracy and characteristics of the approaches used to diagnose tuberculosis in childhood. Hesseling et al. [8] analysed the current approaches to the diagnosis of childhood tuberculosis and concluded the urgent need to develop new diagnostic approaches for childhood tuberculosis. Carreira et al. [15] analysed the WHO, Kenneth-Jones and Keith-Edwards criteria using the Instituto de Pediatria e Puericultura Martagão Gesteira criteria as the gold standard. The Keith-Edwards showed higher values of sensitivity and specificity (84% sensitivity and 97% specificity). However, as this score used response to treatment with non-tuberculosis antibiotics, using this score delays tuberculosis diagnosis. Van Rheenen [21] analysed this score in a high HIV prevalence area, finding 88% sensitivity, 25% specificity, 55% positive predictive value, and 67% negative predictive value. The International Union Against Tuberculosis Lung Disease (IUATLD) point scoring system showed less than 70% sensitivity and specificity and a positive predictive value of 60 – 70% [22].

The scoring system used in Brazil was based on the other scores and on the author’s personal experience in the assistance of tuberculosis patients. The cut-off points used are empiric [7]. The adaptation was based on a decision not to require bacteriology (which is negative in most childhood cases and requires a laboratory and an expert) and not to require a response to non-tuberculosis treatment, which would delay the tuberculosis treatment. It was also validated in a hospital setting using gastric aspirate results as the gold standard: in this case sensitivity was 89% and a specificity of 86% [9]. These results are unlikely to be generalizable to outpatients.

Sant’Anna et al. [16], in a case-control study using a gastric lavage culture that is positive to Mycobacterium tuberculosis as the gold standard, report that the sensitivity of the score ranged from 58% to 89% and the specificity from 98% to 86%, with cut-offs of respectively ⩾40 or ⩾30.

Another study in a reference center in Bahia, Brazil, Sant’Anna et al., using as the gold standard radiologic aspect, clinical, epidemiological and follow-up data, report the use of the scoring system as an auxiliary element to diagnose tuberculosis in culture-negative children and adolescents [17].

Pedrozo et al. [18] also conducted a study to determine the efficacy of the Brazilian scoring system. It was a cross-sectional study performed in a pediatric pulmonology clinic, and a clinical diagnosis of tuberculosis was used as the reference standard. They concluded that the scoring system was valid for the diagnosis of pulmonary tuberculosis. However, in their study, there was a non-independent assessment of test and reference standard, the follow-up period was limited to 30 days, and they did not express accuracy through the usual parameters (sensitivity and specificity, positive and negative predictive values, or positive and negative diagnostic likelihood ratios).

This study showed that, using the point scoring system with 30 points as the cut-off, 80% (95%-CI: 70–90) of children with tuberculosis will be correctly diagnosed: a score higher than 30 provides a factor of 2.6 in favor of tuberculosis, whereas a score lower than 30 yields a factors of 3.4 (1/0.29) in favor of no tuberculosis. The pre-test probability of tuberculosis of 35% changes to 59% in case of a high score (> than 30), reflecting the informativeness of the scoring system and its potential impact on the diagnosis of childhood tuberculosis by the health care system.

HIV co-infection, if present, would interfere in the evaluation of the scoring system. Children and adolescents were not tested for HIV. However, the incidence of AIDS in Brazil, in 2003, was 5.15 per 100,000 inhabitants in the age group 0–4 years, 1.86 per 100,000 inhabitants for those in the age group 5–12 years, and 2.94 per 100,000 inhabitants for those in the age group 13–19 years [23]. Thus, it is not likely that it would influence the results of this study.

The results of this study favor the recommendation of the use of this point scoring system in Brazil and similar sites. The values of sensitivity and specificity are similar to those reported in the literature. In this setting, the cut-off of 30 points can be used to define tuberculosis diagnosis. However, the discriminatory power of the point scoring system may vary across settings as the frequency of different signs and symptoms vary as patients presenting to healthcare facilities with different levels of complexity differ. Therefore, it is recommended that this phase III study be replicated in different settings attended by patients.

5. Conclusion

The results of this study support the use of this point scoring system in Brazil and similar sites with the cut-off of 30 points. However, as the discriminatory power of the point scoring system may vary across settings, it would be useful to replicate this study in different settings.

Acknowledgements

The authors thank all the staff of the Reference Center and Hospital of Infectious Disease Clementino Fraga João Pessoa-Paraíba-Brazil. One author was partially supported by CNPq (Scholarship 308311/2009-4 to R.A.A.X.).

References

[4]D Shingadia and V Novelli, Diagnosis and treatment of tuberculosis in children, LANCET Infec Dis J, Vol. 3, 2003, pp. 624-32.
[5]BRASIL. Ministério da Saúde. Secretaria de Políticas de Saúde. Departamento de Atenção Básica, Manual Técnico para o Controle da Tuberculose: cadernos de atenção básica Ministério da Saúde, Secretaria de Políticas de Saúde, 6ª ed, Ministério da Saúde, Brasília, 2002. Departamento de Atenção Básica
[6]AL Kritisk, GR Souza, and MB Conde, Tuberculose na Infância, AL Kritisk, GR Souza, and MB Conde (editors), Tuberculose: do ambulatório a enfermaria, 3ª ed, Atheneu, São Paulo, 2005, pp. 171-80. chap. 10
[8]AC Hesseling, HS Schaaf, RP Gie, JR Starke, and N Beyers, A critical review of diagnostic approaches used in the diagnosis of childhood tuberculosis, Int J Tuber Lung Dis, Vol. 6, No. 12, 2002, pp. 1038-45.
[9]CTS Orfaliais, Avaliação do sistema de pontuação para o diagnóstico de tuberculose pulmonar na infância, Universidade Federal Fluminense de, Medicina, 2002. Dissertação (Mestrado).
[11]G Stegen, K Jones, and P Kaplan, Criteria for guidance in the diagnosis of tuberculosis, Pediatrics, Vol. 83, No. 2, 1969, pp. 260-3.
[12]PB Fourie, PJ Becker, F Festenstein, GB Migliori, J Alcaide, M Antunes, et al., Procedures for developing a simple scoring method based on unsophisticated criteria for screening children for tuberculosis, Int J Tuber Lung Dis, Vol. 2, No. 2, 1998, pp. 116-23.
[13]PM Nair and EA Philip, Scoring system for the diagnosis of tuberculosis in children, Indian Pediatr, Vol. 18, 1981, pp. 299-303.
[16]CC Sant’’Anna, CTS Orfalliais, March M De FP, and MB Conde, Evaluation of a proposed diagnostic scoring system for pulmonary tuberculosis in Brazilian children, Int J Tuber Lung Dis, Vol. 4, 2006, pp. 463-5.
[17]CC Sant’Anna and M Santos, Diagnosis of pulmonary tuberculosis by score system in children and adolescents: a trial in a reference center in Bahia, Brazil BJID, Vol. 8, 2004, pp. 305-10.
[18]C Pedrozo, CC Sant’Anna, MFBP March, et al., Efficacy of the scoring system, recommended by the Brazilian Ministry of Health, for the diagnosis of pulmonary tuberculosis in children and adolescents, regardless of their HIV status, J Bras Pneumol, Vol. 36, No. 1, 2010, pp. 962-98.
[19]DI Sackett and RB Hones, The architecture of a diagnostic research, BMJ, Vol. 321, 2002, pp. 539-41.
[20]JA Knottnerus, The evidence base of clinical diagnosis, BMJ Books, BMA House, London, 2002.
[21]P Van Rheenen, The use of the paediatric tuberculosis score chart in a HIV-endemic área, TMIH, Vol. 7, No. 5, 2002, pp. 435-40.
Journal
Journal of Epidemiology and Global Health
Volume-Issue
4 - 1
Pages
29 - 34
Publication Date
2013/11/15
ISSN (Online)
2210-6014
ISSN (Print)
2210-6006
DOI
10.1016/j.jegh.2013.10.002How to use a DOI?
Copyright
© 2013 Ministry of Health, Saudi Arabia. Published by Elsevier Ltd.
Open Access
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Cite this article

TY  - JOUR
AU  - Constantino Giovani Braga Cartaxo
AU  - Laura C. Rodrigues
AU  - Carolina Pinheiro Braga
AU  - Ricardo Arraes de Alencar Ximenes
PY  - 2013
DA  - 2013/11/15
TI  - Measuring the accuracy of a point system to diagnose tuberculosis in children with a negative smear or with no smear or culture
JO  - Journal of Epidemiology and Global Health
SP  - 29
EP  - 34
VL  - 4
IS  - 1
SN  - 2210-6014
UR  - https://doi.org/10.1016/j.jegh.2013.10.002
DO  - 10.1016/j.jegh.2013.10.002
ID  - BragaCartaxo2013
ER  -