What is a research paper in computer science? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated?
And now?
This is NOT a lecture !
This is NOT a lecture !
I apologize not to be an epistemologist
I am just a practitioner of computer science that have co-authored some research papers and read/evaluated several hundreds ones
This open discussion is absolutely NOT objective
Goal : open the windows and discuss how we ACTUALLY work (and not how we should work…)
All questions, remarks, contradictions are welcome!!!
Disclaimer
Disclaimer
Discovery vs. invention
What is a research result in computer science?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated?
And now?
My feeling is that there are two main ways of practicing science : discovery vs. invention
My feeling is that there are two main ways of practicing science : discovery vs. invention
Biologists, physicists, chemists, researchers in psychology… are discoverers
(at least, software) Computer scientists, researchers in nanotech, researchers in process engineering or in industrial engineering… are inventors
However inventing often needs a priori discoveries (e.g., of your target/domain of interest); discovering often needs inventing (e.g., measuring instruments)
Mathematicians… are mathematicians :-)
Understanding the world : what are atoms constituted of, why a disease in inherited, why do people have dreams, etc.
Understanding the world : what are atoms constituted of, why a disease in inherited, why do people have dreams, etc.
Understanding means first asking questions, then observing, inquiring, modeling, evaluating
At the end of the research process, one has an answerto the initial question (question of research)
An answer is the most often not definitive. It is an explanation of a small piece of the natural world under some hypotheses
Famous recent example: the “proof of the existence” of Higgs’ boson at CERN/LHC
Software computer science produces inventions
Software computer science produces inventions
Computers do not exist by themselves. They have been created by human beings => there is nothing to discover in a computer or in a software
The objective of research in CS is “just” to make computers/computer networks and computer applications more efficient, more easy to use, more reliable, more powerful… i.e. more useable/useful
As a consequence, a research result in (software) CS has no (real) intrinsic value. It has only the value that the research community and/or the society gives to it. A useless invention has no value!
As a consequence, a research result in (software) CS has no (real) intrinsic value. It has only the value that the research community and/or the society gives to it. A useless invention has no value!
This has to be moderated when one considers research in fundamental and theoretical CS where people study formal problems => context actually close to research in maths
This has also to be moderated when one considers social- or bio- or something-inspired CS where researchers attempt to understand and model an “external” (social or biological or…) phenomenon
Disclaimer
Disclaimer
Discovery vs. invention
What is a research result in computer science?
What is a research paper in computer science? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated?
And now?
The endless debate about fundamental research, basic research, applied research, finalized research…
The endless debate about fundamental research, basic research, applied research, finalized research…
A first and definitive answer: a result has some scientific value as soon as it has been accepted by a scientific publication (a scientific committee), i.e., research is a social activity (hence, fashions, trends…)
A second answer: a result has some scientific value if it interests people and if it is novel (original) (and if it has some genericity) i.e., refer to the scientific method
The most difficult (and so the most interesting) problems have often been raised by applications, e.g. medical applications. Investigating these problems have allowed going through theoretical scientific developments
Disclaimer
Disclaimer
Discovery vs. invention
What is a research result in computer science?
What is a research paper in computer science? General structure(s) of a research paper in CS
Goal of the discussion: analyzing your contribution/proposal
Wrt to your initial objective/motivation
Wrt to the state of the art
Wrt the experiments you have run… and have not run
Wrt the implementation choices you have made
Wrt genericity
A discussion is not a conclusion
The discussion is often distributed over the paper ( :-( ?)
Do not believe authors: make your personal discussion!
A fourth obvious section : your contribution
A fourth obvious section : your contribution
Must be “sound”
Must be comprehensive and complete
Should allow people to reproduce the experiments you use
Must be self-contained
Kill ambiguities
Inventing some “thing” that interests nobody has no real sense
Inventing some “thing” that interests nobody has no real sense
Inventing some “thing” that addresses a problem that has an existing satisfactory solution has a limited interest
To be interesting, a work/paper should address a problem :
With a high (scientific or economic or social or…) value
And no (perfect) solution
(already noted) A paper should list the criteria wrt which authors consider that their proposal is to be evaluated
The often dark side of research in CS
The often dark side of research in CS
A first reason : cultural/educational : computer scientists have an unsatisfactory education in statistics/experiment planning/performance evaluation
However, things become better :-)
A second reason : real life experiments are often intractable because comparing two inventions is often intractable
A second reason : real life experiments are often intractable because comparing two inventions is often intractable
Concurrent software are often not available
Concurrent complex systems/softwares cannot be re-develop from scratch at a reasonable price
Evaluation criteria are different
Users of your software are reluctant to spend/waste their time on alpha versions
It often exists no benchmarks
Research works often address very small range/specific issues that are part of a complex systems but integration is often not realized
Research teams have neither the manpower nor the competence to develop “products” and limit their development to demonstrator or prototypes that cannot be tested by real end users
However experiments are mandatory to evaluate the pertinence of your work
However experiments are mandatory to evaluate the pertinence of your work
Experiments should allow
To evaluate the performance/pertinence/efficiency of your proposal wrt your initial objective and wrt all the possible contexts (not only in the best case)
“Science” definitely condemns treachery and falsification: modifying the result of an experiment is beyond the “red line”
Experimental measurements that are presented in a paper are (supposed to be) true (I personally have always supposed it)
The question is about omitted/”forgotten” results
The question is about uncertainty and statistical errors on the measurements
The question is about some generalization of results/experiments
From a scientific point of view, a negative result is often as interesting as a positive result
However, as CS has to deal with invention, a negative result is usually considered as a depreciated fact
So the temptation to “avoid” including negative measurements often exists:
So the temptation to “avoid” including negative measurements often exists:
“This measurement is not really important but as I do not have enough space to explain it, it is more important, from a scientific point of view (of course), to discuss in deep other more important experiments”
“The situation that leads to this bad result never happens in the real life”
“With a slight modification of the prototype, it is obvious that one would never get such a result, so let us focus on the results concerning the core of the prototype”
etc.
are reasons that a scientist may use to self-persuade him that presenting this experiment is not pertinent
Where is the yellow line? Where is the red line?
Where is the yellow line? Where is the red line?
Always
analyze if experiments are exhaustive
If not, always analyze if the missing experiments are important to evaluate the performance of the prototype
and try to infer what could be the behavior of the prototype wrt these experiments
Disclaimer
Disclaimer
Discovery vs. invention
What is a research result in computer science?
What is a research paper in computer science? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is a research paper evaluated?
And now?
Is the topic of the paper in the domain of interest of the conference/journal?
Is the topic of the paper in the domain of interest of the conference/journal?
