"Methodologies" for research in computer science Lionel Brunie LIRIS lab National Institute of Applied Sciences (INSA) Lyon, France Lionel.Brunie@insa-lyon.fr
Laboratoire d’InfoRmatique de l’Image et des Systèmes d’Information (LIRIS)
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated ?
And now ?
Disclaimer
This is NOT a lecture !
I apologize not to be an epistemologist
I am just a practitioner of computer science that have co-authored ~100 research papers and read/evaluate several hundreds ones
This open discussion is absolutely NOT objective
Goal : open the windows and discuss how we ACTUALLY work (and not how we should work…)
All questions, remarks, contradictions are welcome !!!
Menu
Disclaimer
Discovery vs. invention
What is a research result in computer science ?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated ?
And now ?
Discovery vs. invention
My feeling is that there are two main ways of practicing science : discovery vs. invention
Biologists, physicists, chemists, researchers in psychology… are discoverers
Computer scientists, researchers in nanotech, researchers in process engineering or in industrial engineering… are inventors
Mathematicians… are mathematicians :-)
Discovering (just two words)
Understanding the world : what are atoms constituted of, why a disease in inherited, why do people have dreams, etc.
Understanding means first asking questions, then observing, inquiring, modeling, evaluating
At the end of the research process, one has an answer to the initial question
An answer is the most often not definitive. It is an explanation of a small piece of the natural world under some hypotheses
Inventing
Software computer science produces inventions
Computers do not exist by themselves. They have been created by human beings => there is nothing to discover in a computer or in a software
The objective of research in CS is “just” to make computers and computer networks more efficient more easy to use, more reliable, more powerful… i.e. more useable/useful
As a consequence, a research result in CS has no (real) intrinsic value. It has only the value that the research community and/or the society gives to it. A useless invention has no value !
Menu
Disclaimer
Discovery vs. invention
What is a research result in computer science ?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated ?
And now ?
A difficult answer…
The endless debate about fundamental research, basic research, applied research, finalized research…
A first and definitive answer : a result has some scientific value as soon as it has been accepted by a scientific publication (a scientific committee)
A second answer : a result has some scientific value if it interests people and if it is novel (original) (and if it has some genericity)
The most difficult (and so the most interesting) problems have often been raised by applications, e.g. medical applications. Investigating these problems have gone through theoretical scientific developments.
Menu
Disclaimer
Discovery vs. invention
What is a research result in computer science ?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated ?
And now ?
Structure of a research paper in CS
Introduction-Motivations
State of the art
Proposal
Experiments
Discussion
Conclusion and future work
2 (+1) key sections (1/3)
Having 1 very good idea in a scientist’s life is… very good ; having two is exceptional
One paper => one problem and at most one (two) contribution(s) to defend
A first key section : the state of the art
A state of the art must not be “flat” or general purpose
It must be focused on your scientific and applicative targets and “critical and compared”
Critical : the contribution/limitations of each cited paper should be analyzed wrt your problem
Compared : papers should be organized in taxonomies
The state of the art establishes/describes the scientific basis wrt which your work should be compared, your // will be evaluated
Ideally, all pertinent papers should be analyzed. Practically, it is often untracktable
“Making the bibliography” is a key component of the scientific process
A paper should list the criteria wrt which authors consider that their proposal is to be evaluated
Experiments (1/2)
The often dark side of research in CS
A first reason : cultural/educational : computer scientists have an satisfactory education in statistics/experiment planning/performance evaluation
A second reason : real life experiments are often intractable because comparing two inventions is often intractable
Concurrent software are often not available
Concurrent complex systems/softwares cannot be re-develop from scratch at a reasonable price
Evaluation criteria are different
Users of your software are reluctant to spend/waste their time on alpha versions
It often exists no benchmarks
Research works often address very small range/specific issues that are part of a complex systems but integration is often not realized
Research teams have neither the manpower nor the competence to develop “products” and limit their development to demonstrator or prototypes that cannot be tested by real end users
Experiments (2/2)
However experiments are mandatory to evaluate the pertinence of your work
Experiments should allow
To evaluate the performance/pertinence/efficiency of your proposal wrt your initial objective and wrt all the possible contexts (not only in the best case)
To evaluate the influence of the various parameters on the performance
Experimental conditions must be completely explicited
Menu
Disclaimer
Discovery vs. invention
What is a research result in computer science ?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated ?
And now ?
Reading a paper as an investigation
A key behavior : criticism
A first obvious step : analyze what is written : is the proposal scientifically sound, are the performances good, is the problem really important, is the state of the art exhaustive, etc.
A second step : analyze what is not written
Black boxes and assumptions
Computer systems/applications/softwares are too complex/too large so that a single team/paper can address the entire problem
+ inputs : it exists/will exist (soon) some “system” can can provide your prototype with the data/information it needs
+ outputs : the data produced by your prototype could be/will (soon) be used by the end-user side of the global system
Assumptions are not always explicit
Always ask if these assumptions are valid/realistic, what do they imply
Analyzing experiments (1/2)
The border line
“Science” definitely condemns treachery and falsification : modifying the result of an experiment is beyond the “red line”
Experimental measurements that are presented in a paper are (supposed to be) true (at least I have supposed it)
The question is about omitted/”forgotten” results
From a theoretical scientific point of view, a negative result is often as interesting as a positive result
However, as CS has to deal with invention, a negative result is commonly considered as a depreciated fact
Analyzing experiments (2/2)
So the temptation to “avoid” including negative measurements often exists : “this measurement is not really important but as I do not have enough space to explain it, it is more important, from a scientific point of view (of course), to discuss in deep other more important experiments”, “The situation that leads to this bad result never happens in the real life”, “With a slight modification of the prototype, it is obvious that one would never get such a result, so let us focus on the results concerning the core of the prototype”, etc. are reasons that a scientist may use to self-persuade him that presenting this experiment is not pertinent
Where is the yellow line ? Where is the red line ?
Always analyze if experiments are exhaustive. If not, always analyze if the missing experiments are important to evaluate the performance of the prototype and try to infer what could be the behavior of the prototype wrt these experiments
Menu
Disclaimer
Discovery vs. invention
What is a research result in computer science ?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated ?
And now ?
Evaluation of a research paper
Is the topic of the paper in the domain of interest of the conference/journal ?
(Is the bibliography complete ?)
Is the contribution original ?
Is the paper technically sound/correct ?
Is the paper easy to read ?
Is the contribution significant ?
What is the level of expertise of the evaluator / what is the confidence of the evaluator in his judgement
Discussion within the program or scientific committee
Menu
Disclaimer
Discovery vs. invention
What is a research result in computer science ?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes
How is research evaluated ?
And now ?
What conclusion ?
Read, read, read, read !
Be positively critical
Never admit supposed evidences. Always doubt (Descartes)
