A good start is half the battle won.
“People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying “no” to 1,000 things.” – Steve Jobs
Pick what you want to work on and how to start right is extremely important.
To evaluate your ideas before a further investigation, I like to refer to Prof. Kevin McGee‘s 8 fundamental questions. Note that these questions are developed to evaluate a paper submission, but I think they are equally valuable for screening your ideas as well.
1. What is the Problem or Question the author is attempting to solve or answer?
2. Is the main Problem or Question important enough to warrant study?
- 10 years (or 5 years) impact test. This test is often used by my star colleague Yu Haifeng (http://www.comp.nus.edu.sg/~yuhf/). Before Haifeng starts any projects, he checks if the idea will still be relevant after 10 years. He refuses to work on anything that fails this test.
- However, a project with 10-year impact may also be very difficult to do; therefore, not always suitable for “novice” Ph.D. students who need to PUBLISH PAPERS to LEARN ABOUT RESEARCH. A too difficult/ambitious first project may be harmful to them in establishing confidence in research. Thus, you might want to consider the following two tests instead.
- Get a job test. The central idea behind this test is that: when you presented this project to a good company, will they hire you? Will they be excited? If the answer is no, you may not want to work on it. The Ph.D. program is only a transitional period before your real career. You need to prepare yourself for the job market, and you only have 5 years. Chances are, you can only work on 3 major projects, so every project counts. You have to spend your time wisely. However, similar to the last test, you may not have the skills to finish an impressive project in the beginning. In such cases, it is ok to work on an easier/less impressive project as your first project. However, within your Ph.D. journey, you want to accomplish at least 1 project that can get you a job, so I recommend that you need to pass this test at least once.
Note: quality is much more important than quantity. Finish a project that will be remembered by most people is much more valuable than finishing 10 projects that no one remembers. - Average 4 and above test. This is the last, most basic test. To perform this test on an idea, you first imagine that you have finished working on the project and have written a paper. The work has been PERFECTLY executed, and the paper is FULLY polished. The paper has been submitted to a top conference/journal in your field (such as CHI or UIST). Now imagine you are the reviewer, and what average score do you expect to get? Will you get an average score of 4 and above? If you are not sure about it, then I suggest to drop the idea. If a perfectly written paper on this idea is still on the borderline, then the chance of it getting rejected is too high. My advice: you probably don’t want to work on such ideas unless it is extremely easy to do.
Note: as a graduate student, you may not know whether your idea will get an average score of 4 and above, but your adviser should know, so ask him/her for suggestions.
Caution: sometimes a good idea may be hard to sell in the academy community, so an idea may pass one of the first two tests, but does not pass the last one. In such cases, I will still work on it.
3. What is the author’s contribution to knowledge about solving the Problem or answering the Question? Important: a “contribution” is a contribution to knowledge, so evaluators need to ask “What do readers learn from reading this paper?”
- To understand contributions in HCI, I recommend read the following paper: HCI Research as Problem Solving (https://dl.acm.org/doi/10.1145/2858036.2858283).
4. Is the contribution important or significant ?
- Is the contribution generally relevant? Does it impact more than just a few people (who is interested in the contribution?). And: does it answer or solve more than just a few specific cases of the overall question or problem?
- Is the contribution an important advance over what was known before? (how much do they care about it?)
5. Is the contribution original? An original contribution to knowledge means: readers of the paper will learn something that they cannot learn anywhere else.
6. How do readers know that it is original? It is not enough for a reader to believe (or “know”) that the work is original, the author must clearly identify “related work” and indicate what makes his contribution different.
WARNING! “Related work” is work that has tried to either a) answer the same (or similar) question, or b) solve the same (or similar) problem.
- Answering questions 5 and 6 requires a literature review. A rule of thumb: spend at least 3 days looking for all the possible related work before making a decision. It’s absolutely painful if you find a piece of related work so close to what you want to do after you have already spent a lot of time into it.
7. Can readers trust the validity of the contribution?
- Does the author motivate and document an appropriate method for arriving at results? (This is what the author did: that is, how the author attempted to solve a particular problem or answer a question – and why the author chose the particular method(s) used.)
- Do the results seem believable, significant, relevant, and well-documented? (This is what happened as a result of following the particular method(s).)
- Does the author do an appropriate analysis of those results? (This is the author’s reasoning about what the results mean.)
8. Is the contribution appropriate for a specific discipline (or conference or journal)?
In summary, the evaluation of HCI research involves three main things: is the contribution to knowledge important, original, and trustworthy?
You could try to find these ideas by:
1) Build on the latest research results. For example, one can look for the best papers from a recent conference, read them carefully, and identify their weaknesses or limitations. If one can solve one of such weaknesses or limitations, it’s likely that you are onto a new worthwhile idea. The logic is simple: the best papers from a recent conference are most likely about a significant, original problem. Solving problems/issues they can’t solve are likely to lead to new significant, original research.
2) Combine complementary existing ideas. An example of that is Elastic Hierarchy (https://ieeexplore.ieee.org/document/1532129): it combines a node-link diagram with Treemap to form a new type of visualization technique.
3) Borrow from another field. Often a new technique or approach from another field can be adapted to your field. HCI is a multi-discipline field that can learn from many other fields such as psychology, biology, physics, electronic engineering, etc. A not-so-original idea in one field can be surprisingly innovative when applied in another field.
Image credit: Alpha Stock Images – http://alphastockimages.com/
Photo by Eva Bronzini from Pexels