Introduction

Physicians have many questions when seeing patients. On the other hand, physicians have limited time and resources to search for answers to these questions. HERMES applies natural language processing approaches to automatically generate answers in response to ad-hoc questions posed by physicians. HERMES has the promise to provide physicians the best clinical evidence extracted from primary literature within the time frame demanded in clinical settings. Figure 1 shows the flowchart of HERMES.

Figure 1: Flowchart of HERMES

Extracting information needs from ad-hoc questions is the first component of HERMES. We analyze questions by Question Classification, Query Generation and Query Expansion.

We applied supervised machine-learning approaches (e.g., naive Bayes and logistic regression) for question classification and query generation. We used Weka package for the machine learning. The training data is over 4000 clinical questions maintained by NLM.

Utilities

Question Browsing

[Top]

You can sort, browse and search the clinical questions we downloaded from NLM and NLH

Query-Term Generation

[Top]

It is important to identify "important" topic terms or query terms, and then use those query terms to retrieve documents that potentially answer the question posed by a user. First we identify all the noun phrases in a question, we then weight each noun phrase using the following methods to determine query terms.

Inverse Document Frequency (IDF): The higher the IDF value, the more likely the term is a query term.
Length of the candidate phrase: the longer the term, the more important it may be.
Position of the candidate query term: important query terms may appear towards the end of a question.
Subject/object: a term may be important if it is an object.
Map to the UMLS concepts: if a term can be mapped to a UMLS concept, then the term is more likely to be query term.
Semantic Types: certain semantic types (e.g., "Diseases or Syndrome") determine the importance of a candidate term.

Evaluation:

We perform 10-fold cross validation.

Results:

Following is the score using different features, mainly by logistic regression, also used one Naiive Bayes for algorithm references.

	precision	stdev	recal	stdev	f1	stdev
Logistic(IDF)	66.53%	0.15%	68.24%	0.34%	67.37%	0.24%
Logistic(IDF+LEN)	68.81%	0.26%	73.09%	0.50%	70.89%	0.37%
Logistic(IDF+Len+Pos)	68.68%	0.24%	71.36%	0.46%	70.00%	0.34%
Logistic(IDF+Len+Pos+S/O)	69.08%	0.23%	73.44%	0.46%	71.19%	0.32%
Logistic(IDF+Len+Pos+SemT)	79.30%	0.27%	78.28%	0.34%	78.78%	0.26%
Logistic(IDF+Len+Pos+SemT+S/O)	79.58%	0.28%	77.94%	0.38%	78.75%	0.31%
Logistic(IDF+Len+SemT)	79.30%	0.41%	78.42%	0.25%	78.85%	0.28%
Logistic(IDF+Len+SemT+S/O)	79.64%	0.35%	78.10%	0.32%	78.86%	0.25%
Logistic(IDF+Pos+SemT)	78.24%	0.42%	77.39%	0.39%	77.81%	0.34%
Logistic(IDF+Pos+SemT+S/O)	78.23%	0.41%	77.24%	0.37%	77.73%	0.32%
Logistic(Len+Pos+SemT)	78.73%	0.22%	78.24%	0.47%	78.49%	0.29%
Logistic(Len+Pos+SemT+S/O)	78.92%	0.29%	78.16%	0.49%	78.54%	0.29%
nb(IDF+Len+Pos+SemT)	78.37%	0.26%	74.39%	0.47%	76.33%	0.28%
Logistic(S/O)	52.82%	0.13%	84.26%	0.44%	64.93%	0.23%
Logistic(O)	52.42%	0.08%	88.87%	0.28%	65.94%	0.14%
Logistic(SemT)	76.60%	1.31%	74.48%	1.56%	75.50%	0.44%
Logistic(Len)	64.56%	0.15%	71.18%	0.47%	67.71%	0.29%
Logistic(Pos)	49.62%	0.22%	51.09%	0.34%	50.34%	0.27%

Question Classification

[Top]

Rationale:

Each clinical question can be classified into a general category such as "etiology" and "diagnosis." Question Classification automatically assigns general categories to an ad-hoc clinical question. NLM has defined a total of 12 categories.

What you can do on this site:

you can input an ad-hoc medical question and click classify button to get its category automatically assigned by our system. You can also click "classify" on the right of one question from question browsing interface to do classification.

Method and Evaluation

We applied supervise machine-learning approaches (e.g., naive Bayes and SVMs) using the 4654 questions annotated by the NLM. 10-fold cross validation was performed. Table 2 shows the results. We demon online question classification. Question classification is a computationally intensive task. To simplify the task, the online question classification classifies a question into "diagnosis","treatment_and_prevention","etiology","pharmacological","management," five most popular question categories, and "Others." The features we explored include words, part-of-speech (POS), the UMLS concepts and semantic types (CSTY).

General Topics

Bag-of-Words

With Stemming

With Feature Selection

(top-2000 features)

Words+ Bigrams

Words+ Bigrams+POS

Words+Bigrams

+CSTYs+ Stemming

Words+Bigrams+ POS+CSTYs+ Stemming

Device

Diagnosis

Epidemiology

Etiology

History

Management

Pharmacological

Physical Finding

Procedure

Prognosis

Test

Treatment & Prevention

56.90%

73.68%

70.58%

79.22%

54.29%

68.44%

82.56%

71.74%

70.39%

72.95%

79.06%

68.01%

64.96%

73.75%

68.47%

81.59%

59.23%

68.12%

82.97%

72.38%

71.34%

74.44%

80.62%

68.82%

62.35%

75.15%

65.83%

78.22%

51.30%

67.96%

82.90%

72.72%

69.18%

68.39%

78.95%

69.78%

61.72%

75.88%

67.93%

82.43%

53.76%

71.38%

84.04%

71.14%

66.57%

69.16%

79.17%

70.30%

61.07%

75.23%

67.96%

79.68%

57.91%

71.35%

83.81%

69.58%

65.35%

73.84%

78.18%

69.55%

73.07%

76.78%

72.23%

80.38%

67.71%

71.11%

89.25%

77.75%

80.45%

74.25%

83.04%

71.56%

71.21%

77.17%

70.26%

82.64%

61.70%

70.98%

88.71%

76.67%

80.32%

74.26%

82.37%

70.46%

Average

70.65%

72.22%

70.23%

71.12%

71.13%

76.47%

75.56%

Information Retrieval

[Top]

Once we generate a set of query terms, then we will apply the query terms for information retrieval. You can search across the author , journal title. abstract fields. We indexed the entire MEDLINE with the open-source Solr.

Acknowledgments

We acknowledge the support from the National Library of Medicine to Hong Yu, grant number 1R01LM009836-01A1.
We also thanks the creator of many open source or academic software packages and data (Seam, MMTx, UMLS, Weka, Lucene/Solr and WordNet). Especially thanks James (Jim) G. Mork from MMTx, Mark Hall from Weka, and Karen Steely from NLM clinical question site for their technique supports.

About us

[Top]

Hong Yu's group in University of Wisconsin Milwaukee.

See Members

References

Ely JW, Osheroff JA, et al: Analysis of questions asked by family doctors regarding patient care. BMJ 1999;319:358-361.
Yu H, Lee M, et al: Development, implementation, and a cognitive evaluation of a definitional question answering system for physicians. J Biomed Inform 2007;40:236-251.
Hong Yu, Yong-gang Cao.Automatically Extracting Information Needs from Ad Hoc Clinical Questions. AMIA 2008 Symposium Proceedings pp.96-100