Empirical studies have been conducted in research for a long time. While many evaluations rely on techniques like interviews or questionnaires, any hypothesis about the usage of computer programs and tools has to be evaluated with more advanced methods which offer a greater degree of control and observation. These methods imply a longer setup phase and are generally better suited for smaller experiments under laboratory conditions.
With our new concept we try to shift such evaluations to the cloud by providing preconfigured virtual machines to each participant. The goals of this concept are to provide a framework with which it should be possible to easily and quickly conduct studies without the need of building up an own one consuming a large amount of time. Another huge benefit is that while the evaluation is conducted online it also can be crowdsourced. The pros and contras of crowdsourcing can be found online.
We implemented this concept in a framework called CUPS. The core elements of the implementation are shown in the diagram above. We prepared a case study to evaluate the viability of such studies with CUPS and count on the help of many volunteers. Please help us with our research. Thanks!