Constructing Ontology of Brain Areas and Autism to Support Domain Knowledge Exploration and Discovery

Liang Hong; Haoshuai Xu; Xiaoyue Shi

doi:10.2991/ijcis.d.210203.005

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Volume 14, Issue 1, 2021, Pages 834 - 846

Constructing Ontology of Brain Areas and Autism to Support Domain Knowledge Exploration and Discovery

Authors

Liang Hong¹^{, *}, Haoshuai Xu², Xiaoyue Shi²

¹School of Information Management, Wuhan University, Wuhan, Hubei, China; Center for Studies of Information Resources, Wuhan University, Wuhan, Hubei, China

²School of Information Management, Wuhan University, Wuhan, Hubei, China

^*Corresponding author. Email: hong@whu.edu.cn

Corresponding Author

Liang Hong

Received 30 September 2020, Accepted 31 January 2021, Available Online 10 February 2021.

DOI: 10.2991/ijcis.d.210203.005 How to use a DOI?
Keywords: Ontology; Ontology construction method; Brain area; Autism; Knowledge discovery
Abstract: Medical studies have confirmed the causal relationship between autism and brain areas. Such relationship can effectively promote the early diagnosis and timely intervention of autism. However, existing experiment-driven methods discovering such relationships are costly while machine-learning-based methods are ineffective, because they do not fully utilize the domain knowledge. In this paper, we propose a reasoning-reuse method to construct a Brain Areas-Autism (BAA) ontology to support the domain knowledge discovery, i.e., discovering inherent relationships between autism and brain areas based on BAA ontology. In our method, domain experts first design the schema of the ontology. Then, we use NLP techniques to extract and fuse knowledge from scientific literatures. Rule-based reasoning is performed to expand the scale of ontology. Finally, the ontology is evaluated using the qualitative and quantitative analysis. This paper constructs the BAA ontology with 929 entities and 1129 relationships. Based on this ontology, 130 potential relationships between brain areas and autism were inferred by rule-based reasoning. Experiments demonstrate that the proposed reasoning-reuse method can effectively construct BAA ontology which supports intelligent and efficient knowledge discovery and exploration in domain research.
Copyright: © 2021 The Authors. Published by Atlantis Press B.V.
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

With the continuous emergence of big medical data and the rapid development of artificial intelligence technology, medical research has become more and more “intelligent.” As the core part of medical intelligence, ontology is the unified representation of domain concept system. Ontology facilitates the sharing and linking of medical information and effectively support intelligent applications. In recent years, ontologies of general medical fields, such as open biomedical ontology (OBO) [1], generalized architecture for languages, encyclopedias, and nomenclatures in medicine (OpenGALEN) [2], have promoted the utilization of medical knowledge.

In the field of brain area and autism, brain biological factors are one of the important causes of autism. Some studies [3–5] have confirmed the causal relationship between autism and structural and neurological abnormalities in some brain areas. Such relationship can effectively promote the early diagnosis and timely intervention of autism.

Existing clinical cohort research methods [5,6] discover the medical knowledge by domain experts and experimental study based on medical images. The experimental procedure of this method is complete, and the obtained research results are comparatively accurate. However, the number of brain areas is large, and relationships between concepts of brain areas and autism are complex. Therefore, it's costly to apply clinical cohort research method to the domain of brain area and autism. Some recent studies [7,8] have used machine learning methods to discover new relationships from massive medical images. These methods improve the efficiency of new relationship discovery. However, machine learning models are usually “black box models,” i.e., the learning results are lack of interpretability. This compromises the accuracy and usefulness of knowledge discovery results. Therefore, it is necessary to construct a domain ontology to help revealing complex relationships and discovering new relationships between brain areas and autism.

In this paper, we study the problem of constructing medical ontology effectively and efficiently to organize medical knowledge and discover new knowledge. However, the construction of domain ontologies especially ontologies focusing on specific medical fields faces new challenges. The sheer scale of big medical data requires efficiency, while the inherent complexity of domain knowledge requires accuracy of ontology construction method. In addition, the authority and correctness of medical knowledge is crucial in the field of medicine. Manual ontology construction methods such as skeletal methodology [9] and TOVE [10] do not support the expansion and update of ontology and cannot scale up to big data. Automatic ontology construction methods such as the five-step cycle method [11,12] and cyclic acquisition process [13] have the problem of insufficient accuracy especially in the field of medicine which has a very complex concept system. Ontology construction methods used in medical domain such as the seven-step method [14] lack the necessary step of ontology evaluation.

In order to solve the above challenges, this paper proposes a construction method for medical ontology. Specifically, the main contributions of this research include the following aspects: (1) This paper proposes reasoning-reuse method to construct medical ontologies. Reasoning-reuse method first designs schema to ensure the accuracy of the ontology and then uses rule-based reasoning to expand the scale of ontology. This method also uses the evaluation feedback mechanism to iteratively refine the ontology. (2) We use the reasoning-reuse method to construct the brain areas--autism ontology (BAA ontology) for the first time. The ontology defines the concepts and relationships between brain neuroscience and autism, realizing the systematic and structural organization and representation of domain knowledge. A set of reasoning rules on the BAA ontology are designed based on the professional knowledge of medical literature. 130 potential relationships between brain areas and autism were inferred. These potential relationships give clues to the existing research and therefore improve the efficiency and effectiveness of discovering new relationships. (3) We use the quantitative and qualitative methods to evaluate the BAA ontology construction. Experimental results demonstrate that the BAA ontology presents richer domain knowledge than existing autism ontologies and can support the domain research.

2. RELATED WORK

2.1. Ontology Construction Methods

The skeletal methodology [9] describes the guidelines and basic processes of ontology construction, including three steps of determining the purpose and scope, constructing the ontology and evaluation. TOVE [10] creates the informal description of the ontology first, and then transforms the description into a formal knowledge logic model and evaluates the integrity of the ontology. The seven-step method [14] is a practical ontology construction method proposed for the medical field. Its construction process includes determining the field and scope, reusing existing ontology, listing important terminologies and defining class, hierarchy, attributes, and instances. However, the above methods do not consider the self-expansion and evolution of the ontology. The five-step cycle method [11,12] is an ontology construction method used for semantic web ontology learning. It includes ontology reusing, extracting, pruning, refinement, and application. These 5 steps can be repeated to achieve the update of the ontology. Cyclic acquisition process [13] is a method to obtain domain ontology directly from text through concept and relationship learning. It includes 5 steps of selecting data source, concept learning, field focusing, relationship learning, and evaluation. The five-step cycle method and the cyclic acquisition can realize the self-expansion of the ontology, but they have the problem of insufficient accuracy during the construction process because of the lack of schema. The comparison of these methods with our work is shown in Table 1.

Methods	Schema Design	Self-expansion	Evaluation
Reasoning-reuse (our work)	Yes	Yes	Yes
Skeletal methodology	Yes	No	Yes
TOVE	Yes	No	Yes
Seven-step	Yes	No	No
Five-step cycle	No	Yes	No
Cyclic acquisition	No	Yes	Yes

Table 1

Comparison of ontology construction methods.

Our previous reasoning-reuse method [15] preliminarily proposed the idea of dynamically constructing the ontology by reusing knowledge reasoning results. This article has made big improvements in the human–computer interaction process and tools and realized the larger medical ontology construction and richer knowledge discovery results.

2.2. Medical Ontology

Recently, medical ontologies have been constructed to represent and organize the knowledge in different medical fields. Mccray et al. [16] constructed the ontology of autism spectrum disorder phenotype. They grouped and clustered the items assessed of 24 instruments used to screen or diagnostic autism and summarized 283 concepts related to autism phenotype. This ontology can provide access and inquiry to specific phenotypic data for autism. Kostyuk et al. [17] manually labelled 274 abstracts by experts to identify 30 kinds of terminologies on language disorder that appear in the autism medical literature. They subsequently organized these terminologies as the ontology of language impairment in autism. However, the existing autism ontologies are small in number and scale and lack of autism knowledge associated with the brain area.

In terms of open biomedical field, the Unified Medical Language system [18] established a large-scale medical terminology system covering various medical disciplines such as clinical medicine, basic medicine, pharmacy, biology, medical management, etc. This system has played a significant role in regulating medical terminology and medical classification, but it does not involve medical facts. The OBO [1] integrates data generated in biomedical research for ease of use, including Gene Ontology, Sequence Ontology, and so on. The Generalized Architecture for Languages, Encyclopedias and Nomenclatures in Medicine (OPENGALEN) [2] integrated diverse clinical medical knowledge, such as human anatomy, pathophysiology, surgery, disease, and pharmacy [19]. The ontology of basic theory knowledge in TCM [20] sorted out the knowledge of TCM cognition method, TCM physiology, TCM pathology, syndrome differentiation, and so on. The Hepatitis Ontology [21] contained 55 concepts, 39 attributes, and realized the information retrieval of liver disease. The ontology for assessing health information needed during pregnancy [22] collected terms from social media data and the categories in the literature on pregnancy information. The developed ontology included 241 classes and 788 synonyms. During the COVID-19 pandemic, COVID-19 Surveillance Ontology [23] was developed to support surveillance of COVID-19 in primary care settings. The classes of entities include terms for classifying means of exposure, methods of testing and diagnosis, and subsequent courses of action such as isolation and contact tracing.

2.3. Knowledge Reasoning Based on Ontology

Knowledge reasoning based on ontology is an important way to discover implicit knowledge. Sesen et al. [24] built reasoning rules based on the EU's established drug manufacturing standards and applied them to their previously constructed pharmaceutical ontology, significantly reducing compliance time in the pharmaceutical process and reducing pharmaceutical costs. Kafali et al. [25] constructed an ontology reasoning component based on existing ontology, which integrated patient information from different sources into a same diabetes platform to support the diagnosis and management of diabetes mellitus. Arndt et al. [26] built an intelligent nurse call system through rules-based reasoning. Liu et al. [27] built a database of diabetes diagnosis rules with 28 reasoning rules based on the diabetes ontology they previously established. They discovered potential knowledge in the field of diabetes using semantic reasoning, and achieved the support for diabetes screening, diagnosis, drug recommendation and referral services. Yang et al. [28] built an ontology-based reasoning system for municipal solid waste risk response. They used the reasoning ability of Semantic Web Rule Language (SWRL) to construct a system of rule reasoning for risk transformation.

3. REASONING-REUSE METHOD FOR ONTOLOGY CONSTRUCTION

Since medicine is a highly specialized and complex subject, in reasoning-reuse method, we built domain ontology in a top-down manner.

Because of the complication and profession of medical knowledge, extracting concepts and relationships directly from the data source will lead to an inaccurate ontology. Therefore, the reasoning-reuse method takes schema design as the first step to provide key concepts of the ontology and guide the subsequent data acquisition and instance extraction. Different from existing ontology construction methods, our method uses rule-based reasoning to discover the implicit relationships between concepts, so that the self-expansion and dynamic update of the ontology can be achieved. Finally, the quantitative evaluation is used to inspect the classes with insufficient relationships or instances in the entire ontology. The discovered implicit relationships and the evaluation results are reused and feedback to guide the improvement of the ontology. The framework of reasoning-reuse method is shown in Figure 1.

3.1. Schema Design

The schema design is divided into two steps:

We choose the Web Ontology Language (OWL) to formalize the schema of BAA ontology.

For example, the following defines the Cerebrum, which is a subclass of Brain Area.

Conceptualization

Conceptualization is a process by which experts identify important concepts and relationships of the field knowledge. This process ensures that the subsequent ontology construction steps can be performed under the guidance of a correct medical conceptual framework.

Domain experts initially identify, select, and classify the important and basic concepts and relationships. For example, the schema of BAA ontology is shown in Figure 2. The core concepts of BAA ontology include: Brain Area, Brain Abnormality and ASD (Autistic Spectrum Disorder, Autism) Symptom. Experts in the field of neuroscience and autism define the relationships between these concepts based on objective medical facts. For instance, a specific brain area of autistic patients in a resting state or a specific task state generates a brain abnormality (including functional connectivity abnormality with another different brain area and activation abnormality), causing the body fail to respond normally, so the patients exhibit symptoms of autism. The brain area that has the abovementioned effects on autism should be related to autism. In addition, these brain areas are related to specific autism symptoms. The brain areas, brain abnormality, and autism symptoms are classified into subclass hierarchy according to certain criteria. The following is the description of each concept and its subclasses.
- Brain Area: A hierarchical structure formed by dividing the human brain according to specific classification criteria.
- Brain Abnormality: Activation abnormality and functional connectivity abnormality that occur in the human brain.
- ASD Symptom: Symptoms and characterization of autistic patients, classified according to the autism symptoms description of international standard autism screening and diagnosis tools
Formalization

After the conceptualization of domain knowledge, we should choose a proper ontology language to formalize the schema.

<owl:Class rdf:ID=“Cerebrum”>

<rdfs:subClassOf rdf:resource=“Brain_Area”/>

</rdfs:subClassOf>

</owl:Class>

After the formalization of the concepts, the relationships between concepts need to be formalized as well.

The following defines the relationship “relatedTo,” which indicates the relationship between Brain Area and ASD Symptom. The domain of this relationship is the Brain Area. The range of this relationship is the ASD Symptom.

<owl:ObjectProperty rdf:ID=“relatedTo”>

<rdf:type rdf:resource=http://www.w3.org/2002/07/owl#FunctionalProperty/>

<rdfs:domain rdf:resource=“#Brain_Area”/>

<rdfs:range rdf:resource=“#ASD_Symptom”/>

</owl:ObjectProperty>

Through the above method, the concepts and relationships are formalized. The schema of BAA ontology is organized into a machine-readable form. In the process of formalizing with OWL, concepts and relationships are defined separately, which makes it easy for anyone to define a relationship at any time and assign it to a concept. OWL ensures the flexibility and scalability of formalization.

3.2. Data Initialization

After the step of schema design, we collect and process the required data according to the schema. The purpose of data initialization is to obtain plenty of medical sources required for ontology construction from multi-source heterogeneous data, and then transform them as forms available to the ontology. The step of data initialization includes the selection of the data source and the pre-processing of the data.

3.2.1. Data source selection

The selection of the data source is to determine which medical data sources are required. The amount of medical data that can be used to build ontology is large and varied. Due to the limited availability of structured data sources in the field of medicine, most domain knowledge is contained in a large amount of unstructured medical literature. As shown in Table 2, we select the following data source to construct the BAA ontology.

Domain Knowledge	Data Source	Type
Brain area	Netter's Atlas of Neuroscience [29]	Structured data
Brain area	Anatomical Automatic Labeling (AAL) [30]	Structured data
Autism symptoms	Autism Spectrum Disorder Phenotype Ontology [16]	Structured data
Relationships between brain areas and autism symptoms	6506 Medical Articles from the Literature Database	Unstructured data

Table 2

Selected data sources for BAA ontology.

In terms of the concepts of brain area, this study utilized Netter's Atlas of Neuroscience [29] and Anatomical Automatic Labeling (AAL) [30] to comprehensively and systematically represent the conceptual hierarchy of brain areas. In terms of the concepts of autism symptoms, this study selected the autism symptom terminologies in the existing autism spectrum disorder phenotype ontology [16], which are derived from the description of autism symptoms in 24 internationally recognized standardized autism screening and diagnostic instruments after classification and integration. In terms of the relationships between brain areas and autism symptoms, this study used the lexicon containing brain area names as the search keywords, and then retrieved articles which contain the keywords of autism and one or several of these brain area names. We crawled 6506 medical articles from the large-scale biomedical literature database (PubMed). These articles contain the knowledge of relationships between brain areas and autism found so far. We extract knowledge, i.e., concepts and their relationships, from these unstructured literature as the basis for BAA ontology construction.

3.2.2. Data preprocessing

As shown in the Table 2, most selected data source is unstructured literature. These unstructured data cannot be used to construct the BAA ontology directly. These unstructured literature in pdf format need to be preprocessed. Data preprocessing refers to some processing of the medical data before the ontology construction.

3.3. Instance Construction

According to the schema in the 3.1, we recognize and extract the entities and relationships from the preprocessed medical literature data in 3.2. We use three dictionaries (brain areas, brain abnormalities, and ASD symptoms) that contains a large number of keywords and synonyms of concepts. Firstly, we use the brain areas dictionary and ASD symptoms dictionary to match sentences in the data source. Sentences containing both a brain area keyword and an ASD symptom keyword were matched. Then, we use the brain abnormalities dictionary to match these sentences again. The matched sentences are considered to be very likely to contain knowledge of the relationships between brain areas and autism. Finally, the manually reviewed relationships are added to the ontology. In this step, we convert the knowledge in the medical literature into concepts and the relationships between them in the ontology.

We adopted the idea of human–computer interaction to construct our BAA ontology. To meet the requirement of both high quality and low cost, we proposed a human–computer interaction instance construction mechanism. According to the schema of ontology, we used machine learning models to automatically extract the knowledge from a large number of unstructured texts. The experts were invited to verify and label the extracted knowledge. The eligible knowledge will be added to the BAA ontology. In this way, we could make maximum use of the medical big data, while ensuring the quality of the BAA ontology so that it can meet the accuracy requirements of the medical field.

As shown in Figure 3, instance construction consists of four steps:

Entity extraction. In this step, we used BiLSTM+CRF to extract brain areas and autism symptom in medical literature associated with autism [31]. (Please find more details in https://github.com/HaoshuaiXu/BrainArea-Autism-Ontology)
Relation extraction. In this step, we used attention-based BiLSTM to extract relationship and generate RDF output [32].
Entity fusion and disambiguation. In this step, we removed the duplicate entities and relationships through graph embedding.
Manual review. In this step, those entities and relationships which have high confidence score or approved by experts could be added to the BAA ontology.

Through the above process, we extracted 929 entities and 1129 relationships from 6506 medical literatures and added them to the BAA ontology. The results of BAA ontology construction are shown in Table 3.

Category	Amount	Category	Amount

Entities	929	Relationships	1129
Brain area	760	Brain area—brain abnormality (generate)	178
Brain abnormality	9	Brain abnormality—ASD symptom (cause)	178
ASD symptom	160	Brain area—ASD symptom (related to)	753
		Functionally connected with	20

Table 3

The BAA ontology construction results.

An example of entities and relationships in BAA ontology is shown in Figure 4. The entity frontal lobe is an instance of Cerebrum in the schema. According to the knowledge in the medical literature, we extracted 7 types of relationship about this entity. The frontal lobe generates reduced activation and causes the social cognition symptom of ASD. The frontal lobe is functionally connected with the anterior temporal cortices. This kind of connectivity abnormality caused the learning difficulties symptom of ASD. In this way, we realized the representation and organization of the knowledge in brain areas and autism field.

3.4. Ontology Expansion

The step of ontology expansion is an important part of implementing ontology self-expansion and dynamic update. We use rule-based reasoning to discover the implicit relationships between concepts.

In the step of rule-based reasoning, we generated two basic reasoning rules based on domain knowledge, including Relevance Transfer Rule and Loop Derivation Rule. We use description logics to define them:

Relevance Transfer Rule:

∃ connetWithBrainAreax, BrainAreay⊓ relateTo BrainAreay, Symptoms⇒relateTo BrainAreax, Symptoms

BrainAreax means x is an individual of Brain Area. Symptoms means s is an individual of Symptom.
connectWithx,y means x has functional connectivity with y.
relateTox,y means x is relate to y.

As shown in Figure 5, Relevance Transfer Rule means if there are two brain areas x and y which have functional connectivity with each other and x is related to an autism symptom, then y is related to the same autism symptom as well.

Loop Derivation Rule:

∃ connetWithBrainAreax, BrainAreay⊓ relateTo BrainAreax, Symptoms⊓ relateTo BrainAreay, Symptoms⇒connection AbnormalityBrainAreax,BrainAreay

(x, y) means x has functional connectivity abnormality with y.

As shown in Figure 6, Loop Derivation Rule means if two brain areas which have functional connectivity with each other are relate to the same autism symptom, they have functional connectivity abnormality with each other.

Due to the characteristics of the rule-based reasoning, the basis of reasoning was presented at the same time, which guaranteed the interpretability of reasoning conclusions, and ensured the accuracy and reliability of the reasoning results.

These knowledge discovery results were added to the ontology. Such self-expansion further improved the quality and completeness of the ontology.

3.5. Ontology Evaluation

We used the quantitative evaluation method to estimate the rationality of the BAA ontology schema and instances. Firstly, the comparison of the evaluation results of the BAA ontology and other existing autism ontologies can help us verify the advantages of using reasoning-reuse method to build the medical ontology. Secondly, the comparison of the evaluation results of each class within the BAA ontology allowed us to find the classes with insufficient relationships or instances. These evaluation results will tell us which classes should be focused and which relationships and instances should be supplemented in the next iteration of the BAA ontology construction.

We chose 4 indexes of the quantitative index evaluation framework OntoQA [33] to evaluate the BAA ontology constructed by the reasoning–reuse method. These 4 indexes can quantitatively evaluate the rationality of ontology schema and instances and the ability to organize and represent medical knowledge. The specific evaluation indexes are shown in Table 4.

Category of Index	Evaluation Index	Formula	Meaning
Schema index	Relationship richness	RR=\|p\|\|SC\|+\|p\|	The ontology with more relationships besides inheritance has higher relationship richness.
	Attribute richness	AR=\|att\|\|C\|	The greater the average number of attributes per class, the richer the knowledge conveyed by the ontology.
	Inheritance relationship richness	IRS=∑CiϵC\|HCC1,Ci\|\|C\|	The ontology with many inheritance levels but few subclasses is vertical, otherwise, it is horizontal.
Instance index	Average population	AP=\|I\|\|C\|	The average distribution of instances across all classes. The larger AP is, the more abundant instances are extracted.

Table 4

The quantitative evaluation indexes.

(|SC|: number of inheritance relationships.) (|p|: number of relationships other than inheritance.) (|att|:number of attributes of all classes.) (|C|: number of classes defined in the ontology schema.) (|HCC1,Ci|: number of subclasses C1 of each class Ci.) (|I|: the number of instances of the ontology.)

Firstly, we compared BAA ontology with other two existing autism ontologies—the autism spectrum disorder phenotype ontology [16] and the ontology of language impairment in autism [17]. As shown in Table 5:

Compared with the other two existing autism ontologies, the BAA ontology has the highest relationship richness and attribute richness. This indicates that the BAA ontology contains more information about relationships and attributes of concepts.
The BAA ontology has the lowest inheritance relationship richness among all three ontologies.
The BAA ontology has abundant instances. In summary, BAA ontology constructed by the reasoning–reuse method can represent the complex concepts and relationships in the medical filed.

Evaluation Index	BAA Ontology	ASD Phenotype Ontology [16]	Autism Language Impairment Ontology [17]
Relationship richness	0.47	0	0
Attribute richness	0.07	0	0
Inheritance relationship richness	2.024	3.623	5.833
Average population	84.45	No instance	No instance

Table 5

The comparison of the evaluation results of the BAA ontology and other existing autism ontologies.

Secondly, as shown in Table 6, we compared the evaluation results of each class within the BAA ontology. We found that the vertical characteristic of the knowledge, the attributes and the instances were concentrated in the brain area. According to the feedback of these evaluation results, we should extract more attributes and instances of brain abnormality and ASD Symptom in the next iteration of the BAA ontology construction.

Evaluation Index	Brain Area	Brain Abnormality	ASD Symptom
Relationship richness	0.43	0.5	0.5
Attribute richness	0.21	0	0
Inheritance relationship richness	1.21	2.43	2.43
Average population	152	48.05	53.3

Table 6

The comparison of the evaluation results of each class within the BAA ontology.

4. EXPERIMENT

We conducted the knowledge reasoning experiment to realize the domain knowledge discovery based on the BAA ontology. We realized two aspects of domain knowledge discovery: (i) the implicit relationships between brain area and ASD symptom and (ii) the probable functional connectivity abnormality between brain areas. We also used the ontology visualization to support domain knowledge exploration and reasoning. Finally, we designed a questionnaire evaluation experiment and invited the experts to evaluate the performance of the BAA ontology in knowledge discovery.

4.1. Knowledge Discovery Based on the BAA Ontology Reasoning

We used the rule-based reasoning to discover the potential relationships between brain areas and autism. 130 potential relationships between brain areas and autism were inferred, in which 20 results are implicit relationships between brain area and ASD symptom and 110 results are probable functional connectivity abnormalities between brain areas. These knowledge discovery results can be used for both self-expansion of ontology scale and helping medical researchers to discover new research points.

4.1.1. Discovery of relationships between brain area and ASD symptom

We find 10 relationships between brain area functional connectivity abnormality and ASD symptom based on unstructured medical literatures. Through the rule-based reasoning experiments, we inferred 20 brain areas potentially associated with autism. The experiment results of domain knowledge discovery of relationships between brain area and ASD symptom are shown in Table 7. In the Table 7, the format of existing relationships in the BAA ontology is (x-y)-s, which means the brain areas x and y are functionally connected and associated with the ASD symptom s. The format of implicit relationships inferred by Reasoning is x-s, which means the brain area x is related to the ASD symptom s.

We use the case experiment to verify the usefulness of domain knowledge discovery.

According to the reasoning rule Relevance Transfer Rule in Section 3.4, if the brain areas–extrastriate cortex and superior temporal sulcus are functionally connected and associated with autism, extrastriate cortex can be inferred to be potentially associated with autism. Superior temporal sulcus can be inferred to be potentially associated with autism as well (as shown in the Table 7). In order to verify the usefulness of the reasoning result, we used the “((extrastriate cortex [Title/Abstract]) AND superior temporal sulcus [Title/Abstract]) AND autism [Title/Abstract]” as the search strategy. The strategy retrieves the earliest publication dates of study on the relationship between the functional connectivity of extrastriate cortex and superior temporal sulcus and autism. Similarly, we used the “(extrastriate cortex [Title/Abstract]) AND autism [Title/Abstract]” and ‘(superior temporal sulcus [Title/Abstract]) AND autism [Title/Abstract]’ to retrieve the earliest publication dates of study on the relationships between extrastriate cortex and autism, superior temporal sulcus and autism respectively. The results obtained after the search are shown in Table 7. The earliest document that proves extrastriate cortex and superior temporal sulcus are functionally connected and associated with autism was published in 2002 [34]. The earliest study of the correlation between the extrastriate cortex and autism was published in 2004 [35]. The earliest study of the relationship between the superior temporal sulcus and autism was published in 2006 [36].

We found that 90% of the verification results (the green items in Figure 7) meet the regularity: in the research on the correlation between brain areas and autism, most of studies have earlier found that brain functional connectivity between the two brain areas is related to autism, and later found that one of the brain areas is associated with autism.

We further found that among all the experiment results of domain knowledge discovery of relationships between Brain Area and ASD Symptom (Table 7), there are 4 implicit relationships (②'', ③', ③'' and ⑩⑩') that have not been confirmed by clinical studies. However, as shown in Figure 7, the earliest study publication dates of other relationships corresponding to these 4 relationships (②'', ③', ③'', ⑩') all meet the regularity discussed in the previous paragraph. Therefore, these four implicit relationships can be provided to clinical researchers as the future research direction.

In summary, the proposed domain knowledge discovery based on rule-based reasoning can guide existing research directions on brain area and autism and help discover new relationships.

	Existing Relationships in the BAA Ontology		Implicit Relationships Inferred by Reasoning
①	(extrastriate cortex – superior temporal sulcus)-visual information [34]	①'	extrastriate cortex- visual information [35]
①		①''	superior temporal sulcus- visual information [36]
②	(temporal-parietal occipital junction)-cognitive deficits [37]	②'	temporal-cognitive deficits [38]
②		②''	parietal occipital junction-cognitive deficits
③	(hippocampal temporal-parietal junction)-schizophrenic syndrome [39]	③'	hippocampal temporal-schizophrenic syndrome
③		③''	parietal junction-schizophrenic syndrome
④	(amygdala-ventromedial prefrontal cortex)- habituation [40]	④'	amygdala- habituation [41]
④	(amygdala-ventromedial prefrontal cortex)- habituation [40]	④''	ventromedial prefrontal cortex- habituation [42]
⑤	(prefrontal cortex-the amygdala)-depression [43]	⑤'	prefrontal cortex-depression [44]
⑤	(prefrontal cortex-the amygdala)-depression [43]	⑤''	the amygdala-depression [45]
⑥	(medial temporal-prefrontal areas)-schizophrenia [46]	⑥'	medial temporal-schizophrenia [47]
⑥	(medial temporal-prefrontal areas)-schizophrenia [46]	⑥''	prefrontal areas-schizophrenia [48]
⑦	(cortico-basal ganglia circuit dysfunction)-depression [49]	⑦'	cortico-depression [50]
⑦	(cortico-basal ganglia circuit dysfunction)-depression [49]	⑦''	basal ganglia circuit dysfunction-depression [51]
⑧	(frontal-anterior temporal cortices)-learning difficulties [52]	⑧'	frontal-learning difficulties [53]
⑧		⑧''	anterior temporal cortices-learning difficulties [54]
⑨	(striatal-thalamic circuit)-Bipolar disorder [55]	⑨'	striatal-Bipolar disorder [56]
⑨	(striatal-thalamic circuit)-Bipolar disorder [55]	⑨''	thalamic circuit-Bipolar disorder [57]
⑩	(striatal regions-cingulate cortex)-Bipolar disorder [58]	⑩'	striatal regions-Bipolar disorder
⑩	(striatal regions-cingulate cortex)-Bipolar disorder [58]	⑩''	cingulate cortex-Bipolar disorder [59]

Table 7

The experiment results of domain knowledge discovery of relationships between brain area and ASD symptom.

4.1.2. Discovery of functional connectivity abnormality between brain areas

We conducted experiments and gained 110 pairs of brain functional connectivity potentially associated with the ASD symptoms. The results of the top 20 pairs which have functional connectivity abnormality are shown in Table 8.

Function Connectivity With	Number of Related Symptom
<cingulate_gyrus, Hippocampus>	9
<cingulate_gyrus, Insula>	9
<fusiform_gyrus, Insula>	9
<Thalamus, cingulate_gyrus>	9
<fusiform_gyrus, Amygdala>	9
<Thalamus, fusiform_gyrus>	8
<cingulate_gyrus, Amygdala>	8
<cingulate_gyrus, Caudate>	7
<fusiform_gyrus, Hippocampus>	6
<fusiform_gyrus, cingulate_gyrus>	6
<fusiform_gyrus, Caudate>	6
<Cuneus, cingulate_gyrus>	5
<Cuneus, fusiform_gyrus>	5
<Amygdala_R, Insula>	5
<Lingual_gyrus, cingulate_gyrus>	5
<cingulate_gyrus, Precentral_gyrus>	5
<Fusiform, cingulate_gyrus>	5
<Hippocampus, Amygdala_R>	4
<Hippocampus, left_amygdala>	4
<Left_amygdala, Insula>	4

Table 8

Discovery of functional connectivity abnormality between brain areas.

4.2. Knowledge Exploration Based on the BAA Ontology Visualization

The scale of knowledge about brain area and autism is very large. For constructing BAA ontology, we extracted more than 9,000 medical literature and the number of entities and relationships of BAA ontology is up to more than 2,000. It can be assumed that human perception is not enough to deal with this number of information. In order to make better use of the large amount of information in our ontology for medical researcher, we also designed an ontology visualization system to assist medical research of autism.

In our system, we presented all the knowledge of our ontology in structure of graph. The nodes of the graphs represent concepts of brain-area and autism, and the edges of graphs represent the relationship inside the brain area and autism, and between brain area and autism symptoms. The edges inside brain area have attributes that reflect information about whether the brain functional connectivity is abnormal or not. Users can click graph or query information through inputting text to see the relationship between brain area and autism and can also add nodes and edges in real time to explore or verify possible relationship between brain area and autism. As shown in figure 8, we used this visualization way to help medical researchers more fully utilize information of brain area and autism in our ontology and avoid them getting lost in their exploration.

Besides, our system also has two main features:

Semi-automated knowledge reasoning

As a complement to our ontology expansion, we also provided a semi-automated way to perform knowledge reasoning. As shown in figure 9, this is a crowdsourced, rule-based visualization of knowledge reasoning. We have built into the visualization system an engine for knowledge inference and two concept-level inference rules Relevance Transfer Rule and Loop Derivation Rule, which are mentioned in the paper. We provided an interface for the user to add instance-level inference rules and made them able to get the results of the inference from that rule in real time. For inference rules whose results are valuable, users can choose to save them in the system. Other users can invoke this rule to reproduce the results. We accumulated inference rules through this crowdsourcing approach and realized knowledge communication.
Knowledge Linking

We proposed this feature to help medical researchers validate knowledge they discover or product. We correlated some critical conceptions and relationships of brain area and autism in our ontology with links to their scientific literature and display them in the appropriate places in our visualization system. In this way, we made it possible for users to view the sources of knowledge at any time and validate the knowledge they discover or product, thus helping the medical study of autism.

4.3. Questionnaire Evaluation

We designed a qualitative evaluation experiment and invited experts to assess the performance of BAA ontology in knowledge discovery.

Questionnaire Design

We designed a questionnaire covering 6 assessment factors of knowledge discovery and 3 assessment factors of user experience during the evaluation experiment. For each factor, the experts would be asked to describe their opinions with very low, low, medium, high, or very high. The questionnaire for knowledge discovery evaluation is shown in Table 9.
Evaluation Process

We invited 7 medical experts to accomplish the questionnaires. These experts had not been involved in the development of any of the BAA ontology.

We provided the experts participating in the evaluation with the BAA ontology visualization website and necessary instructions and guidance documents. These instructions explain in detail the basic functions and usage methods of the BAA ontology visualization website.
Evaluation Results

The results of knowledge discovery evaluation are shown as a diverging stacked bar chart in Figure 10. The chart presents the total percentage of positive results (calculated as high + very high), the total percentage of negative results (calculated as low + very low), and the percentage of participants who were neutral (equal to the percentage of the medium), for each evaluation item, in order to show the central tendency in each item.

According to the results, it can be concluded that (i) the BAA ontology performs well in the items of “Explicit Knowledge Accuracy,” “Readability,” and “Understandability.” Most experts also gave positive comments on “Potential Knowledge Richness” and “Potential Knowledge Accuracy.” (ii) The BAA ontology realizes the potential knowledge discovery based on explicit knowledge.

Factor	Type (Very Low, Low, Medium, High, Very High)
Knowledge Discovery
Explicit knowledge richness
Potential knowledge richness
Explicit knowledge accuracy
Potential knowledge accuracy
Explicit knowledge usefulness
Potential knowledge usefulness
User Experience
Readability
Understandability
Usage Willingness

Table 9

The questionnaire for knowledge discovery evaluation.

5. CONCLUSIONS

This paper proposed a semi-automatic ontology construction method (reasoning-reuse method) to construct medical ontologies reliably. Using the reasoning-reuse method, we constructed the brain areas–autism ontology (BAA ontology) which contains 929 entities and 1129 relationships. Based on BAA ontology, we used rule-based reasoning to infer 130 potential relationships between brain areas and autism. Finally, we used the quantitative and qualitative methods to evaluate the BAA ontology construction results. Experimental results demonstrate that the BAA ontology presents richer domain knowledge than existing autism ontologies and can support the autism research in big data era.

In the future, we will continue to (i) improve our design of BAA ontology schema according to our ontology evaluation, in order to accommodate more knowledge of brain area and autism; (ii) discover more relations and rules between brain area and autism through our reasoning based on rules, in order to improve the efficiency of medical research; (iii) improve BAA ontology visualization system and increase more literature of knowledge linking, in order to help medical researcher verify knowledge they get from BAA ontology.

CONFLICTS OF INTEREST

The authors declare of no conflicts of interest.

AUTHORS' CONTRIBUTIONS

Liang Hong contributed to the conception of the study and wrote the manuscript; Haoshuai Xu performed the experiment and wrote the manuscript; Xiaoyue Shi performed the data analyses and wrote the manuscript.

Funding Statement

This paper is supported by National Natural Science Foundation of China No. 72074172, Key Research and Development Program of Hubei Province No. 2020BAB026.

ACKNOWLEDGMENTS

Thanks for Professor Long Lu for his comments and suggestions for the preliminary version. Thanks for all the experts participating the questionnaire.

REFERENCES

1.Wikipedia contributors, OBO Foundry. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=OBO_Foundry&oldid=1000113062

2.Opengalen. http://www.opengalen.org/

3.M. Dapretto, M.S. Davies, J.H. Pfeifer, et al., Understanding emotions in others: mirror neuron dysfunction in children with autism spectrum disorders, J. Nat. Neurosci., Vol. 9, 2006, pp. 28-30.

4.L. Oberman and V.S. Ramachandran, The simulating social mind: the role of the mirror neuron system and simulation in the social and communicative deficits of autism spectrum disorders, J. Psychol. Bull., Vol. 133, 2007, pp. 310-327.

5.T.J. DeVito, D.J. Drost, R.J. Neufeld, et al., Evidence for cortical dysfunction in autism: a proton magnetic resonance spectroscopic imaging study, J. Biol. Psychiatry, Vol. 61, 2007, pp. 465-473.

6.L.M. Oberman, E.M. Hubbard, J.P. McCleery, et al., EEG evidence for mirror neuron dysfunction in autism spectrum disorders, J. Cogn. Brain Res., Vol. 24, 2005, pp. 190-198.

7.X. Guo et al., Diagnosing autism spectrum disorder from brain resting-state functional connectivity patterns using a deep neural network with a novel feature selection method, J. Front. Neurosci., Vol. 11, 2017, pp. 460.

8.G. Deshpande, L. Libero, K.R. Sreenivasan, et al., Identification of neural connectivity signatures of autism using machine learning, J. Front. Hum. Neurosci., Vol. 7, 2013, pp. 670.

9.M. Uschold and M. Gruninger, Ontologies: principles, methods and applications, J. Knowl. Eng. Rev., Vol. 11, 1996, pp. 14-17.

10.M. Grüninger and M.S. Fox, Methodology for the design and evaluation of ontologies., 1995.

11.A. Maedche and S. Staab, Ontology Learning for the Semantic Web, Kluwer Academic Publishers, Boston, MA, USA, 2001.

12.X. Shang, Comparative analysis of foreign ontology construction methods, J. Lib. Inf. Serv., Vol. 56, 2012, pp. 116-119.

13.J. Kietz, R. Volz, and A. Maedche, Extracting a domain-specific ontology from a corporate intranet, in Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop (Lisbon, Portuga), 2000.

14.N.F. Noy and D.L. McGuinness, Ontology development 101: a guide to creating your first ontology, 2001. http://www.ksl.stanford.edu/people/dlm/papers/ontology-tutorial-noy-mcguinness.pdf

15.L. Hong and X. Shi, A method for constructing medical ontology: the case of brain area and autism, J. Inf. Resources Manag., Vol. 10, 2020, pp. 80-90.

16.A.T. Mccray, P. Trevvett, and H.R. Frost, Modeling the autism spectrum disorder phenotype, J. Neuroinformat., Vol. 12, 2014, pp. 291-305.

17.N. Kostyuk, R.D. Isokpehi, R.V. Rajnarayanan, et al., Areas of language impairment in autism, J. Autism Insights, Vol. 2, 2010, pp. 31-38.

18.NLM. UMLS. https://www.nlm.nih.gov/research/umls/

19.Z. Wu, W. Huang, D. Mou, et al., Comparative study on ontology projects in biomedical field, Chinese J. Med. Lib. Inf. Sci., Vol. 19, 2010, pp. 16-19.

20.D. Zhang, Y. Xie, M. Li, et al., Construction of knowledge graph of traditional Chinese medicine based on the ontology, J. Technol. Intell. Eng., Vol. 3, 2017, pp. 35-42.

21.Y. Chen, Study on Hepaptitis Ontology and Semantic Similarity Analysis, Zhejiang University, Zhejiang, China, 2017.

22.J.Y. Lee, An ontology for assessing health information needed during pregnancy, J. Stud. Health Technol. Inf., Vol. 264, 2019, pp. 1520-1521.

23.H. Liyanage, S. de Lusignan, and J. Williams, COVID-19 surveillance ontology.. https://bioportal.bioontology.org/ontologies/COVID19/

24.M.B. Sesen, P. Suresh, R. Bañares-Alcántara, et al., Development of a computer support system for the management of regulatory compliance of pharmaceutical processes, J. Comput. Aided Chem. Eng., Vol. 27, 2009, pp. 2013-2018.

25.O. Kafali, M. Sindlar, T. Weide, et al., ORC: an ontology reasoning component for diabetes, in Proceeding of 2nd International Workshop on Artificial Intelligence and NetMedicine (NetMed) (Beijing, China), 2013.

26.D. Arndt, B.D. Meester, P. Bonte, et al., Ontology reasoning using rules in an eHealth context, N. Bassiliades, G. Gottlob, F. Sadri, A. Paschke, and D. Roman (editors), Rule Technologies: Foundations, Tools, and Applications, Springer, Cham, Switzerland, 2015, pp. 465-472.

27.Z. Liu, C. Xia, L. Huang, et al., Development of diabetes mellitus ontology and realization of semantic inference, Chinese J. Med. Lib. Inf. Sci., Vol. 26, 2017, pp. 7-11.

28.Q. Yang, C. Zuo, X. Liu, et al., Risk response for municipal solid waste crisis using ontology-based reasoning, Int. J. Environ. Res. Pub. Health, Vol. 17, 2020, pp. 3312.

29.L.F. David, K.O. Michael, and E.M. Mary, Netter's Atlas of Neuroscience, Elsevier, 2015.

30.N. Tzouriomazoyer, B. Landeau, D. Papathanassiou, et al., Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain, J. Neuroimage, Vol. 15, 2002, pp. 273-289.

31.Z. Huang, W. Xu, and K. Yu, Bidirectional LSTM-CRF models for sequence tagging, 2015. arXiv preprint arXiv: 1508.01991

32.P. Zhou et al., Attention-based bidirectional long short-term memory networks for relation classification, in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (Berlin, Germany), 2016.

33.S. Tartir, I.B. Arpinar, and M. Moore, OntoQA: metric-based ontology quality analysis, in Proceeding of IEEE Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources (New Orleans, Louisiana, USA), 2005.

34.F. Castelli, C. Frith, F. Happé, et al., Autism, Asperger syndrome and brain mechanisms for the attribution of mental states to animated shapes, J. Brain, Vol. 125, 2002, pp. 1839-1849.

35.Y. Takarae, N.J. Minshew, B. Luna, et al., Pursuit eye movement deficits in autism, J. Brain, Vol. 127, 2004, pp. 2584-2594.

36.M. Zilbovicius, I. Meresse, N. Chabane, et al., Autism, the superior temporal sulcus and social perception, J. Trends Neurosci., Vol. 29, 2006, pp. 359-366.

37.M. Zilbovicius, N. Boddaert, P. Belin, et al., Temporal lobe dysfunction in childhood autism: a PET study, Am. J. Psychiatry, Vol. 157, 2000, pp. 1988-1993.

38.A.E. Pinkham, J.B. Hopfinger, K.A. Pelphrey, et al., Neural bases for impaired social cognition in schizophrenia and autism spectrum disorders, J. Schizophrenia Res., Vol. 99, 2008, pp. 164-175.

39.C.G. Wible, Hippocampal temporal-parietal junction interaction in the production of psychotic symptoms: a framework for understanding the schizophrenic syndrome, J. Front. Hum. Neurosci., Vol. 6, 2012, pp. 180.

40.J.R. Swartz, J.L. Wiggins, M. Carrasco, et al., Amygdala habituation and prefrontal functional connectivity in youth with autism spectrum disorders, J. Am. Acad. Child Adolescent Psychiatry, Vol. 52, 2013, pp. 84-93.

41.D.G. Amaral and B.A. Corbett, The amygdala, autism and anxiety, J. Novartis Found. Symp., Vol. 251, 2003, pp. 177-187.

42.A. Di Martino and F.X. Castellanos, Functional neuroimaging of social cognition in pervasive developmental disorders: a brief review, J. Ann. New York Acad. Sci., Vol. 1008, 2003, pp. 256-260.

43.R.J. Davidson and H.A. Slagter, Probing emotion in the developing brain: functional neuroimaging in the assessment of the neural substrates of emotion in normal and disordered children and adolescents, J. Ment. Retard. Dev. Disabil. Res. Rev., Vol. 6, 2000, pp. 166-170.

44.P. Xu, A. Chen, Y. Li, et al., Medial prefrontal cortex in neurological diseases, J. Physiol. Genomics, Vol. 51, 2019, pp. 432-442.

45.A.C. Felix-Ortiz and K.M. Tye, Amygdala inputs to the ventral hippocampus bidirectionally modulate social behavior, J. Neurosci., Vol. 34, 2014, pp. 586-595.

46.T. Ohnishi, H. Matsuda, T. Hashimoto, et al., Abnormal regional cerebral blood flow in childhood autism, Brain J. Neurol., Vol. 123, 2000, pp. 1838-1844.

47.A.A. Willette, G.R. Lubach, R.C. Knickmeyer, et al., Brain enlargement and increased behavioral and cytokine reactivity in infant monkeys following acute prenatal endotoxemia, J. Behav. Brain Res., Vol. 219, 2011, pp. 108-115.

48.Z. Abdi and T. Sharma, Social cognition and its neural correlates in schizophrenia and autism, J. CNS Spectrums, Vol. 9, 2004, pp. 335-343.

49.R.T. Peixoto, W. Wang, D.M. Croney, et al., Early hyperactivity and precocious maturation of corticostriatal circuits in Shank3B(-/-) mice, J. Nat. Neurosci., Vol. 19, 2016, pp. 716-724.

50.B.B. Braden, C.J. Smith, A. Thompson, et al., Executive function and functional and structural brain differences in middle-age adults with autism spectrum disorder, J. Autism Res., Vol. 10, 2017, pp. 1945-1959.

51.G. Martella, M. Meringolo, L. Trobiani, et al., The neurobiological bases of autism spectrum disorders: the R451C-neuroligin 3 mutation hampers the expression of long-term synaptic depression in the dorsal striatum, Eur. J. Neurosci., Vol. 47, 2018, pp. 701-708.

52.G.B. Hall, H. Szechtman, C. Nahmias, et al., Enhanced salience and emotion recognition in Autism: a PET study, Am. J. Psychiatry, Vol. 160, 2003, pp. 1439-1441.

53.L.E. Campbell, E. Daly, F. Toal, et al., Brain and behaviour in children with 22q11.2 deletion syndrome: a volumetric and voxel-based morphometry MRI study, Brain J. Neurol., Vol. 129, 2006, pp. 1218-1228.

54.B.N. Anil Kumar, S. Malhotra, A. Bhattacharya, et al., Regional cerebral glucose metabolism and its association with phenotype and cognitive functioning in patients with autism, Indian J. Psychol. Med., Vol. 39, 2017, pp. 262-270.

55.X. Wang, A.L. Bey, B.M. Katz, et al., Altered mGluR5-Homer scaffolds and corticostriatal connectivity in a Shank3 complete knockout model of autism, J. Nat. Commun., Vol. 7, 2016, pp. 11459.

56.T. Bordia, D. Zhang, X.A. Perez, et al., Striatal cholinergic interneurons and D2 receptor-expressing GABAergic medium spiny neurons regulate tardive dyskinesia, J. Exper. Neurol., Vol. 286, 2016, pp. 32-39.

57.F. Cao, J.J. Liu, S. Zhou, et al., Neuroligin 2 regulates absence seizures and behavioral arrests through GABAergic transmission within the thalamocortical circuitry, J. Nat. Commun., Vol. 11, 2020, pp. 3744.

58.Y. Takarae, N.J. Minshew, B. Luna, et al., Atypical involvement of frontostriatal systems during sensorimotor control in autism, J. Psychiatry Res., Vol. 156, 2007, pp. 117-127.

59.M. Brüne, A. Schöbel, R. Karau, et al., Neuroanatomical correlates of suicide in psychosis: the possible role of von Economo neurons, PloS One, Vol. 6, 2011, pp. e20936.

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Journal: International Journal of Computational Intelligence Systems
Volume-Issue: 14 - 1
Pages: 834 - 846
Publication Date: 2021/02/10
ISSN (Online): 1875-6883
ISSN (Print): 1875-6891
DOI: 10.2991/ijcis.d.210203.005 How to use a DOI?
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

ris enw bib

TY  - JOUR
AU  - Liang Hong
AU  - Haoshuai Xu
AU  - Xiaoyue Shi
PY  - 2021
DA  - 2021/02/10
TI  - Constructing Ontology of Brain Areas and Autism to Support Domain Knowledge Exploration and Discovery
JO  - International Journal of Computational Intelligence Systems
SP  - 834
EP  - 846
VL  - 14
IS  - 1
SN  - 1875-6883
UR  - https://doi.org/10.2991/ijcis.d.210203.005
DO  - 10.2991/ijcis.d.210203.005
ID  - Hong2021
ER  -

download .riscopy to clipboard