Named Entity Recognition (NER) is a fundamental task in information extraction that locates the mentions of named entities and classifies them (e.g., person, organization and location) in unstructured texts. The NER task has traditionally been solved as a sequence labeling problem, where entity boundaries and category labels are jointly predicted. Chinese NER is more difficult to process than English NER. Chinese language is logographic and provides no conventional features like capitalization. In addition, due to a lack of delimiters between characters, Chinese NER is correlated with word segmentation, and named entity boundaries are also word boundaries. However, incorrectly segmented entity boundaries will cause error propagation in NER. For example, in a particular context, a disease entity “思覺失調症” (schizophrenia) may be incorrectly segmented into three words: “思覺” (thinking and feeling), “失調” (disorder) and “症” (disease).
In the digital era, healthcare information-seeking users usually search and browse web content in click-through trails to obtain healthcare-related information before making a doctor’s appointment for diagnosis and treatment. Web texts are valuable sources to provide healthcare information such as health-related news, digital health magazines and medical question/answer forums. Domain-specific healthcare information includes many proper names, mainly as named entities, such as “葡萄糖六磷酸鹽去氫酶” (Glucose-6-Phosphate Dehydrogenase; G6PD), “電腦斷層掃描” (computed tomography; CT), and “靜脈免疫球蛋白注射” (intravenous immunoglobulin; IVIG). In summary, Chinese healthcare NER is an important and essential task in natural language processing to automatically identify healthcare entities such as symptoms, chemicals, diseases, and treatments for machine reading and understanding.
Following the ROCLING-2022 shared task focused on Chinese healthcare NER, we organize a MultiNER-Health shared task for multi-genre NER in the healthcare domain. In this shared task, we have three genres:
1. Formal texts (FT): this includes health news and articles written by professional editors or journalists.
2. Social media (SM): this contains texts from crowed users in medical question/answer forums.
3. Wikipedia articles (WA): this free online encyclopedia includes articles created and edited by volunteers worldwide.
Named entities may be used in different word forms in other genres. For example, “後天免疫缺乏症候群” (Acquired Immunodeficiency Syndrome; AIDS) is commonly used as a spoken language form “愛滋病” in the medical forums. On the other hand, “甘油三酯” is a different usage referred to as “三酸甘油酯” (triglyceride; TG) in Wikipedia.
A total of 10 entity types are described and some examples are provided in Table I for Chinese healthcare NER. In this task, participants are asked to predict the named entity boundaries and categories for each given sentence. We use the common BIO (Beginning, Inside, and Outside) format for NER tasks. The B-prefix before a tag indicates that the character is the beginning of a named entity and I-prefix before a tag indicates that the character is inside a named entity. An O tag indicates that a token belongs to no named entity. Below are the example sentences.
| Entity Type | Description | Examples |
|---|---|---|
| Body (BODY) |
The whole physical structure that forms a person or animal including biological cells, organizations, organs and systems. | “細胞核” (nucleus), “神經組織” (nerve tissue), “左心房” (left atrium), “脊髓” (spinal cord), “呼吸系統” (respiratory system) |
| Symptom (SYMP) |
Any feeling of illness or physical or mental change that is caused by a particular disease. | “流鼻水” (rhinorrhea), “咳嗽” (cough), “貧血” (anemia), “失眠” (insomnia), “心悸” (palpitation), “耳鳴” (tinnitus) |
| Instrument (INST) |
A tool or other device used for performing a particular medical task such as diagnosis and treatments. | “血壓計” (blood pressure meter), “達文西手臂” (DaVinci Robots), “體脂肪計” (body fat monitor), “雷射手術刀” (laser scalpel) |
| Examination (EXAM) |
The act of looking at or checking something carefully in order to discover possible diseases. | “聽力檢查” (hearing test), “腦電波圖” (electroencephalography; EEG), “核磁共振造影” (magnetic resonance imaging; MRI) |
| Chemical (CHEM) |
Any basic chemical element typically found in the human body. | “去氧核糖核酸” (deoxyribonucleic acid; DNA), “糖化血色素” (glycated hemoglobin), “膽固醇” (cholesterol), “尿酸” (uric acid) |
| Disease (DISE) |
An illness of people or animals caused by infection or a failure of health rather than by an accident. | “小兒麻痺症” (poliomyelitis; polio), “帕金森氏症” (Parkinson’s disease), “青光眼” (glaucoma), “肺結核” (tuberculosis) |
| Drug (DRUG) |
Any natural or artificially made chemical used as a medicine. | “阿斯匹靈” (aspirin), “普拿疼” (acetaminophen), “青黴素” (penicillin), “流感疫苗” (influenza vaccination) |
| Supplement (SUPP) |
Something added to something else to improve human health. | “維他命” (vitamin), “膠原蛋白” (collagen), “益生菌” (probiotics), “葡萄糖胺” (glucosamine), “葉黃素” (lutein) |
| Treatment (TREAT) |
A method of behavior used to treat diseases. | “藥物治療” (pharmacotherapy), “胃切除術” (gastrectomy), “標靶治療” (targeted therapy), “外科手術” (surgery) |
| Time (TIME) |
Element of existence measured in minutes, days, years. | “嬰兒期” (infancy), “幼兒時期” (early childhood), “青春期” (adolescence), “生理期” (on one’s period), “孕期” (pregnancy) |
| Entity Type | Description | Input & Output |
|---|---|---|
| Formal Texts |
Ex 1 | Input: 早起也能預防老化，甚至降低阿茲海默症的風險 Output: O, O, O, O, O, O, B-SYMP, I-SYMP, O, O, O, O, O, B-DISE, I-DISE, I-DISE, I-DISE, I-DISE, O, O, O |
| Ex 2 | Input: 壓力、月經引起的痘痘患者 Output: B-SYMP, I-SYMP, O, B-TIME, I-TIME, O, O, O, B-DISE, I-DISE, O, O |
|
| Social Media |
Ex 3 | Input: 如何治療胃食道逆流症？ Output: O, O, O, O, B-DISE, I-DISE, I-DISE, I-DISE, I-DISE, I-DISE, O |
| Ex 4 | Input: 請問長期打善思達針劑是不是會變胖? Output: O, O, O, O, O, B-DRUG, I-DRUG, I-DRUG, I-DRUG, I-DRUG, O, O, O, O, B-SYMP, I-SYMP, O? |
|
| Wikipedia Articels |
Ex 5 | Input: 抗生素和維生素Ａ酸可用於口服治療痤瘡 Output: B-DRUG, I-DRUG, I-DRUG, O, B-DRUG, I-DRUG, I-DRUG, I-DRUG, B-DRUG, O, O, O, O, O, O, O, B-DISE, I-DISE (“痤瘡” is a formal usage of “痘痘” in the example 2 ) |
| Ex 6 | Input: 抑酸劑，又稱抗酸劑，抑制胃酸分泌，緩解燒心。 Output: B-CHEM, I-CHEM, I-CHEM, O, O, O, B-CHEM, I-CHEM, I-CHEM, O, O, O, B-CHEM, I-CHEM, O, O, O, O, O, B-DISE, I-DISE, O (“燒心” is the spoken language of “胃食道逆流症” in the example 3 ) |
| Training Set | |||
|---|---|---|---|
| Genre | Formal Texts |
Social Media |
Wikipedia Articles |
| #Sentence | 23,008 | 7,684 | 3,205 |
| #Character | 1,109,918 | 403,570 | 118,116 |
| #Named Entities |
42,070 | 26,390 | 13,369 |
| Data Sets | Chinese HealthNER Corpus (Lee and Lu, 2021) |
CHNER Dataset (Lee et al., 2022) |
|
The policy of this shared task is an open test. Participating systems are allowed to use other publicly available data for this shared task, but the use of other data should be specified in the final system description paper.
Testing Set at least 2,000 Chinese sentences per genre will be provided for system performance evaluation.
The performance is evaluated by examining the difference between machine-predicted labels and human-annotated labels. We adopt standard precision, recall, and F1-score, which are the most typical evaluation metrics of NER systems at a character level. If the predicted tag of a character in terms of BIO format was completely identical with the gold standard, that is one of the defined BIO tags, the character in the testing instance was regarded as correctly recognized. Precision is defined as the percentage of named entities found by the NER system that are correct. Recall is the percentage of named entities present in the test set found by the NER system. Different genre will be evaluated independently. The Macro-averaging F1 score among three genres will be used for final ranking in the leaderboard.
| Team | Run# | F1-score (%) | Rank | |||
|---|---|---|---|---|---|---|
| Fornal Texts |
Social Media |
Wikipedia Articles |
Macro- Averaging |
|||
| CrowNER [1] | Run 2 | 65.49 | 69.43 | 73.63 | 69.55 | 1 |
| YNU-HPCC [2] | Run 2 | 61.96 | 71.11 | 72.13 | 68.40 | 2 |
| ISLab [3] | Run 1 | 62.52 | 71.42 | 71.19 | 68.38 | 3 |
| SCU-MESCLab [4] | Run 1 | 62.51 | 71.33 | 70.57 | 68.14 | 4 |
| YNU-ISE-ZXW [5] | Run 3 | 62.79 | 70.22 | 70.37 | 67.79 | 5 |
| LingX [6] | Run 2 | 51.23 | 59.28 | 60.54 | 57.02 | 6 |
| Baseline [7] (Bilstm-CRF) | Word2vec | 60.99 | 67.16 | 67.91 | 65.35 | - |
| BERT | 61.08 | 70.77 | 72.54 | 68.13 | - | |
[1] Yin-Chieh Wang, Wen-Hong Wu, Feng-Yu Kuo, Han-Chun Wu, Te-Yu Chi, Te-Lun Yang, Sheh Chen, and Jyh-Shing Roger Jang. 2023. CrowNER at ROCLING 2023 MultiNER-Health Task: enhancing NER task with GPT paraphrase augmentation on sparsely labeled data.
[2] Chonglin Pang, You Zhang, and Xiaobing Zhou. YUN-HPCC at ROCLING 2023 MultiNER-Health Task: a transformer-based approach for Chinese healthcare NER.
[3] Jun-Jie Wu, Tao-Hsing Chang, and Fu-Yuan Hsu. 2023. ISLab at ROCLING 2023. MultiNER-Health Task: a three-stage NER model combining textual content and label semantics.
[4] Tzu-En Su, Ruei-Cyuan Su, Ming-Hsiang Su, and Tsung-Hsien Yang. 2023. SCU-MESCLab at ROCLING 2023 MultiNER-Health Task: named entity recognition using multiple classifier model.
[5] Xingwei Zhang, Jin Wang, and Xuejie Zhang. 2023. YUN-ISE-ZXW at ROCLING 2023 MultiNER-Health Task: a transformer-based model with LoRA for Chinese healthcare named entity recognition.
[6] Xuelin Wang and Qihao Yang. 2023. LingX at ROCLING 2023 MultiNER-Health Task: intelligent capture of Chinese medical named entities by LLMs.
[7] Lung-Hao Lee, Chien-Huan Lu, and Tzu-Mi Lin. 2022. NCUEE-NLP at SemEval-2022. Task 11: Chinese named entity recognition using the BERT-BiLSTM-CRF model. In Proceedings of the 16th International Workshop on Semantic Evaluation. Association for Computational Linguistics, pages 1597-1602.
We provide a starter kit on our GitHub Repo. It can be used to create the BiLSTM-CRF system for the NER tasks. https://github.com/NCUEE-NLPLab/AITutorial-2022-ChineseNER