Fusion of ALBERT and BiGRU-Attention-CRF Models for Named Entity Recognition in Lakes
1. School of Electronics and Information Engineering, Guangdong Ocean University, Zhanjiang 524088, China
2. School of Chemistry and Environment, Guangdong Ocean University, Zhanjiang 524088, China
| Abstract: | There are many lakes in China and they are widely distributed, so it is important to have comprehensive information on lakes to promote the implementation of major water network projects, ecological and environmental restoration of rivers and lakes, and smart water conservancy construction. This paper uses a deep learning model to identify and extract lake entities, and provides a reference for mining effective lake information from lake text. Firstly, the Named Entity Recognition (NER) method and annotation specification are customized, and a corpus of lake texts is established. Secondly, considering the small size of the self-built corpus and the sparsity of text information may affect the performance of the model and the recognition effect, a lightweight pre-training model (ALBERT) is introduced to generate high-quality word vectors as a two-way gating cycle The unit-conditional random field (BiGRU-CRF) model is used to generate the input layer feature vectors, and the attention mechanism is incorporated into the model to add feature weights to the semantic information of the entities to improve the entity feature extraction effect. Finally, a large number of validation experiments are completed, and the recognition effects of the four deep learning models are compared. The results show that the ALBERT-BiGRU-Attention-CRF model has good recognition effect on the dataset as a whole, with accuracy, recall and F1 reaching 91.26%, 90.38% and 90.81%, respectively. In addition, the model outperforms the other comparison models for the recognition of all four types of entities that occur at high frequencies. |
| Keywords: | NER; ALBERT; BiGRU; Attention; CRF |
| DOI: | 10.57237/j.se.2022.01.003 |
View:
Downloads:
Received: 29 September 2022
Accepted: 21 October 2022
Published Online: 1 November 2022
Funding
| 1. | 国家自然科学基金项目 (42176067) |
| 2. | 广东海洋大学强校工程项目 (Q18307) |
References
| [1] | 第一次全国水利普查公报 [J]. 中华人民共和国水利部公报, 2013 (02): 53-57. |
| [2] | Rau L F. Extracting Company Names from Text [C] // The Seventh IEEE Conference on Artificial Intelligence Application. IEEE, 1991: 29-32. |
| [3] | 李军怀, 陈苗苗, 王怀军, 崔颖安, 张爱华. 基于ALBERT-BGRU-CRF的中文命名实体识别方法 [J]. 计算机工程, 2022, 48 (06): 89-94+106. |
| [4] | 巩敦卫, 张永凯, 郭一楠, 王斌, 樊宽鲁, 火焱. 融合多特征嵌入与注意力机制的中文电子病历命名实体识别 [J]. 工程科学学报, 2021, 43 (09): 1190-1196. |
| [5] | Chen Y, Xu L, Liu K, et al. Event extraction via dynamic multi-pooling convolutional neural networks [C] // Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015: 167-176. |
| [6] | Voorhees E, Harman DK (2006) Trec: experiment and evaluation in information retrieval. J Am Soc Inf Sci Technol 32 (4): 563–567. |
| [7] | Fader A, Zettlemoyer L, Etzioni O. Paraphrase-driven learning for open question answering [C] // Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2013: 1608-1618. |
| [8] | Yao X, Van Durme B. Information extraction over structured data: Question answering with freebase [C] // Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2014: 956-966. |
| [9] | 许宁. 面向旅游领域的智能问答系统设计与实现 [D]. 内蒙古大学, 2021. |
| [10] | Bunescu R, Mooney R. A shortest path dependency kernel for relation extraction [C] // Proceedings of human language technology conference and conference on empirical methods in natural language processing. 2005: 724-731. |
| [11] | Miwa M, Bansal M. End-to-end relation extraction using lstms on sequences and tree structures. In: ACL. 2016: 1105–1116. |
| [12] | Singh S, Riedel S, Martin B, Zheng J, McCallum A. Joint inference of entities, relations, and coreference. In: AKBC. 2013: 1–6. |
| [13] | Han S, Hao X, Huang H. An event-extraction approach for business analysis from online Chinese news [J]. Electronic Commerce Research and Applications, 2018, 28: 244-260. |
| [14] | Upadhyay S, Gupta N, Roth D. Joint multilingual supervision for cross-lingual entity linking [J]. arXiv preprint arXiv: 1809. 07657, 2018. |
| [15] | Li J, Bu C, Li P, et al. A coarse-to-fine collective entity linking method for heterogeneous information networks [J]. Knowledge-Based Systems, 2021, 228: 107286. |
| [16] | 冯钧, 杭婷婷, 陈菊, 王云峰, 王秉发, 张涛. 领域知识图谱研究进展及其在水利领域的应用 [J]. 河海大学学报(自然科学版), 2021, 49 (01): 26-34. |
| [17] | 段浩, 韩昆, 赵红莉, 蒋云钟, 李豪, 毛文山. 水利综合知识图谱构建研究 [J]. 水利学报, 2021, 52 (08): 948-958. |
| [18] | 刘婷, 张社荣, 王超, 李志竑, 关炜, 王枭华. 水利施工事故文本智能分析的BERT-BiLSTM混合模型 [J]. 水力发电学报, 2022, 1 (07): 1-12. |
| [19] | Cao P, Chen Y, Liu K, et al. Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism [C] // Proceedings of the 2018 conference on empirical methods in natural language processing. 2018: 182-192. |
| [20] | Xu C, Wang F, Han J, Li C. Exploiting multiple embeddings for Chinese named entity recognition. In: Proceedings of the 28th ACM international conference on information and knowledge management. 2019: 2269-2272. |
| [21] | Greenberg N, Bansal T, V erga P, McCallum A. Marginal likelihood training of bilstm-crf for biomedical named entity recognition from disjoint label sets. In: Proceedings of the 2018 conference on empirical methods in natural language processing. 2018: 2824–2829. |
| [22] | Lan Z, Chen M, Goodman S, et al. Albert: A lite bert for self-supervised learning of language representations [J]. arXiv preprint arXiv:1909.11942, 2019. |
| [23] | http://www.lakesci.csdb.cn |
| [24] | Ju M, Miwa M, Ananiadou S. A neural layered model for nested named entity recognition. In: Proceedings of the 2018 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, vol 1 (Long Papers). 2018: 1446–1459. |
| [25] | Chung J, Gulcehre C, Cho K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling [J]. arXiv preprint arXiv: 1412. 3555, 2014. |
| [26] | Zhu Q L, Li X L, Conesa A, et al. GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical text. Bioinformatics. 2018, 34 (9): 1547. |
| [27] | WOJEK C SCHIELE B. A dynamic conditional random field model for joint labeling of object and scene classes [C] // Proceedings of the 10th European Conference on Computer Vision, LNCS 5305. Berlin: Springer. 2008: 733-747. |
Join the Editorial Board
We invite active, qualified and high profile scientists and researchers to join as Editorial Board Members.
Join UsBecome a Reviewer
Scholars with a strong interest in reviewing are invited to join the reviewer panel to ensure the quality of the research to be published.
Join Us