Summary of Reviews for Paper of BMC Bioinformatics

The English rephrasing comments are the easiest to deal with. I list all of this kind of comments in this section.

• In section of Abstract: "Admittedly" is required to be changed
• In section of Background: "An example of event" to "of an event"
• In subsection of Data: "abstracts of full articles" is supposed to be replaced by "abstracts or full articles"
• In subsection of Post-Processing: "extra-arguments" is supposed to be split into two words
• In subsection of Related Work: "are ran" should be "run"; "consequential" is required to be changed
• In subsection Of Performance Analysis: "rightfully" is required to be changed

Most of detail enrichment comments need me to add more details about the definitions, methods and hyper-parameters, which are not difficult.

• Abstract
  • "the best results" is not true. The reviewer think it is fair to add FAUST system in comparison according to the task requirements.
• Background
  • "huge amounts of electronic bio-medical documents" is unnecessary imprecise.
    I do not understand what the reviewer want. A precise number of documents?
  • "[bio-medical documents] involve domain-specific […] dependencies" is an unclear claim. The same as the comment above.
  • "15 times faster [than Riedel et al.] " should specify the condition (e.g. omission of feature caching)
  • Data
    • "extracts of articles from PubMed Central": some are only in PubMed, not PMC.
      It is my mistake. I thought they are the same thing by inferring from their names. In fact, PubMed is the super-set of PMC, where articles on PMC can be freely download but only abstracts are accessible for other articles on PubMed.
    • About the discussion of consistency and effect on performance, reviewer suggest to consider contrasting analysis with that presented in "Overview of the ID, EPI and REL tasks of BioNLP Shared Task 2011"
      I read the article suggested, and found a statement "each entity annotation consists of a type". Note that this article is published for BioNLP 2011. It does not explain the inconsistency between BioNLP2009 and 2011.
• Methods
  • Algorithm1G
    • The reviewer suggest to replace "score and label" and "highest score" with more precise descriptions of the steps.
      
  • Fitting the Pairwise Model
    • "two hyper-parameters that are set by cross-validation". More information on selection (e.g. considered values) are required.
    • The method that the author used to deal with the heavy class imbalance should be added.
      We already had cited the paper about the method we used, which is C+/C-.
  • Computational Considerations
    • "SVM-based classification scales well" is unnecessarily imprecise. Efficient linear SVM implementations are invariant to the number of examples at prediction time (the weight vector is explicitely represented)
      I do not understand
  • Pre-Processing & Features
    • Add examples of the two tokenizations
      "inhibit NF-kappaB-dependent pro-inflammatory gene transcription" vs "inhibit NF-kappaB dependent pro-inflammatory gene transcription"
    • Add examples of each feature for clarity.
      Should we add examples in the feature table or in paragraph? The table is not large enough to contain examples whereas enumerating examples for each feature in paragraph would be very ugly to see.
    • Add details of generating IntAct features.
      
    • Identify the "heuristics from the UCLEED system"
      Does this requirement means explaining how to trim the dependency path?
      Reply in reponse
  • Related Work
    • Mention rule-based (e.g. NCBI) and pattern-based (e.g. NICTA) methods.
• Results
  • Add the comparison with the system by Miwa et al (2010), as it reported one of the best performances, particularly with Binding events.
    This system was run on the BioNLP2009 data set, we do not have results on this data set.
  • Add FAUST system into comparison
• Conclusion
  • Provide more detail regarding future work
  • "the best result" is not true (FAUST is the best).

The arguable comments contain something that reviewers do not agree with us. Some of this kind of comments need the supports from new experiments, which could cost certain amount of time. Considering the deadline of the camera version and my defense date, I have to carefully organize the schedule for them. Besides, I doubt about some of these comments.

• Parts of the terminology are confusing. The word "entity" can be interpreted as a term in biology and a term in text processing. Besides, the trigger is not exactly a "named entity".
  I think it is better to replace named entity by text entity and add definition that a text entity is a character chain.
• Methods
  • Algorithm
    • The definition of P_S=(c_i,a_j) is inconsistent because the arguments are initialized by proteins but involve predicted triggers during the process.
      
  • Pre-Processing
    • About the claim "high quality dependency parse trees require a fine grained tokenization", reviewer require us to provide evidence to support this claim.
      It is hard to give a direct evidence of the quality of dependency parse. We can only say that dependency parse with find grained tokenization provides better features for this task. But the quality of dependency parse itself is not defined. To demonstrate the better effect of dependency parse with fine grained tokenization, we have to run experiments with dependency parsing based on coarse tokenization.
    • "trees are finally obtained using ….". Reviewer thought such parses are provided in the supporting data?
      The support data are generated using different tokenization, hence are differ to what we generated.
      emphasize we computed them ourselves.
  • Related work
    • "rich features coding for (trigger, argument) pairs [16] are only used by pipeline models for assigning arguments": this is false at least for the TEES and EVEX systems.
      /This is differ to what I read. TEES has two steps: trigger detection and edge detection. The features of pairs are not used in trigger detection./
      rephrase and explanation for reviewer.
• Results
  • Genia Shared Task 2013
    • "it is difficult to estimate standard errors properly": consider referencing statistical significance results in [1].
      Reply in the response email
    • clarify if EVEX and TEES also use the data-set from BioNLP2011
      They did not talk about this in their papers published in the workshop. I have sent a mail to ask which data-set did they use. However, even if they reply me and say they had used BioLNP2011 data-set, their mail can not act as reference.
  • Genia Shared Task 2011
    • "TEES (no detailed results available)": available on the task web page and in "The Genia Event and Protein Co-reference tasks of the BioNLP Shared Task 2011"
      
  • precision-recall curves with different features
    • Figure 5 in the result is easy to misinterpret. From this curve it seems that V-walk features

are better left out. The authors mention that this is due to the limited size of the test set, but I would like to see a more careful analysis on how this phenomenon scales with the size of the test set. Related to this, the authors can do a more elaborate feature selection step: this is easy to do with SVM, and improved results using feature selection for event extraction have been described before. I would like to discuss about it.
Do not worth experiments. But I have to be aware of feature selection methods (Lasso).

• training duration
  • more theoretical characterization of the computational complexity (Reviewer 1)
    It should be in the section of Methods. I do not understand the reviewer's opinion.
    Explain our limitation with a soft description that claims the uncertainty
  • Scala is slower than C. The training duration may be caused by the programming language. (reviewer 1)
  • efficiency order of pipeline > RUPEE > globally joint system is rough and lack of support. There is no training duration test for pipeline system and only one case for joint system. (reviewer 3)
    20 mins for pipeline couterpart

Requirements in the complement comments need much time. The comments of reviewer 3 are for future job in my opinion.

• Make software usable outside the BioNLP community. Web tool or API that provide biologists to use this method on text data. (reviewer 1)
  Display like http://corpora.informatik.hu-berlin.de
  It will cost a large amount of time to add the protein recognition ourselves.
• Make general framework for all the BioNLP tasks. Will break the constraints in Genia task. (reviewer 3) It is not a requirement for this paper, but for the further work

Comments about format are essential to publish our paper.

• Figures (e.g. figure 5) are hard to interpret with printed out in gray-scale
• BMC Bioinformatics does not allow footnotes and requires that URLs be included in references. Link to BMC format.