EduCloud PaaS versus IaaS Cloud Usage for an Advanced Comput

590IEEE TRANSACTIONS ON EDUCATION,VOL.54,NO.4,NOVEMBER2011 EduCloud:PaaS versus IaaS Cloud Usage

for an Advanced Computer Science Course

Luis M.Vaquero,Member,IEEE

Abstract—The cloud has become a widely used term in academia and the 7d8b26f84693daef5ef73d28cation has not remained unaware of this trend,and several educational solutions based on cloud technolo-gies are already in place,especially for software as a service cloud. However,an evaluation of the educational potential of infrastruc-ture and platform clouds has not been explored yet.An evaluation of which type of cloud would be the most bene?cial for students to learn,depending on the technical knowledge required for its usage,is missing.Here,the?rst systematic evaluation of different types of cloud technologies in an advanced course on network overlays with84students and four professors is presented.This evaluation tries to answer the question whether cloud technologies (and which speci?c type of cloud)can be useful in educational scenarios for computer science students by focusing students in the actual tasks at hand.This study demonstrates that platform clouds are valued by both students and professors to achieve the course objectives and that clouds offer a signi?cant improvement over the previous situation in labs where much effort was devoted to setting up the software necessary for course activities.These results most strongly apply to courses in which students interact with resources that are non-self-contained(e.g.,network nodes, databases,mechanical equipment,or the cloud itself),but could also apply to other science disciplines that involve programming or performing virtual experiments.

Index Terms—Cloud,course,education,evaluation.

I.I NTRODUCTION

H IGHER education is characterized by“the tension be-

tween the offered quality and the drive to provide afford-able higher education to more and more people”[1].Informa-tion technologies(IT)have an increasing role as supporting el-ements to improve the quality(access to more resources that can be globally widespread)and to reduce the cost of resource usage,thus easing the sustainability of education.

Different IT technologies have been proposed for educa-tional support in a variety of courses,such as those in advanced networking and distributed systems.Service-oriented architec-tures(SOAs)[2],[3]and Grid[4],[5]are examples of this. However,they imply a high entry barrier for educators and students who have to learn the new tools.This is a well-known problem related to the usage of IT solutions as education-sup-porting elements[6].Also,SOA and Grid solutions,as with many other computing technologies,require certain hardware and software infrastructure to be setup before they can be used

Manuscript received September30,2010;revised November10,2010;ac-cepted November30,2010.Date of publication January13,2011;date of cur-rent version November02,2011.

The author is with Hewlett-Packard Labs,Bristol BS348QZ,U.K.,and also with Universidad Rey Juan Carlos,Móstoles28933,Spain(e-mail: lmvaquero@7d8b26f84693daef5ef73d28).

Digital Object Identi?er10.1109/TE.2010.2100097for lab assignments.This demands specialized technical skills from both professors and students to set up the experimental environment.These tasks often distract students from the real goals of the course since they are focusing their effort on peripheral tasks more related to system administration than to the course topics.

Cloud computing[7]is a new paradigm for the provision of every network-available resource(X)as a service(XaaS).Al-though there are many de?nitions for the cloud,most of them include these four basic ingredients.

?On-demand resource provisioning:The required resources (machines,online libraries,version control systems,data management APIs,etc.)do not need to be online or de-ployed beforehand.

?Pay-per-use model:Users pay only for the actual usage of resources.

?Quality guarantees:Quality of service is guaranteed by the provider by the usage of service level agreements(SLAs).?Scalability:The“size”of the service dynamically and au-tomatically adapts to the actual demand.

Assuming that all clouds provide these four features,clouds are also usually categorized in three types,depending on the abstraction level of the provided services.

?Infrastructure-as-a-Service(IaaS)clouds supply vir-tual hardware resources such as machines,networks,or storage.A well-known example is Amazon EC2.1Here, users must adapt,deploy,and control all the software stack their applications rely on.

?Platform-as-a-Service(PaaS)clouds provide a container environment for users to run their software compo-nents—for instance,an application container,an online database management system shared with several users, other mechanisms for data persistence to be used by online applications deployed over the platform,etc.Examples are Google App Engine2(GAE)and Microsoft’s Azure.3?Software-as-a-Service(SaaS)clouds give access to typical applications(e-mail clients,etc.)through the network.As such,they are intended for?nal users,not for system devel-opers.The problem of dealing with education-supporting technologies is here reduced to selecting and aggregating in an appropriate order a set of prepackaged software ser-vices,and so this category is not included in this study. These two?rst different levels of clouds are shown in Fig.1. As can be observed,IaaS clouds offer an abstraction layer pre-venting users from dealing with low-level hardware con?gura-tions(but still dealing with the software and operating system, 17d8b26f84693daef5ef73d28/7d8b26f84693daef5ef73d28st viewed December2010.

27d8b26f84693daef5ef73d28/7d8b26f84693daef5ef73d28st viewed December2010.

37d8b26f84693daef5ef73d28/7d8b26f84693daef5ef73d28st viewed December2010.

0018-9359/$26.00?2011IEEE

V AQUERO:EduCloud:PaaS VERSUS IaaS CLOUD USAGE FOR ADV ANCED COMPUTER SCIENCE COURSE

591

Fig.1.Clouds classi?ed according to the abstraction-level of the services they provide.

OS,in their virtual machines,VMs).IaaS clouds can be directly exposed to?nal users(such as Amazon’s EC2),or they can be used by providers of more abstract services to host their ser-vices.Fig.1illustrates how PaaS clouds can be built upon an underlying IaaS cloud.PaaS clouds,in turn,offer services to help developers throughout the whole life cycle of an applica-tion,from design to test and to production.In the examples in Fig.1,it can be observed how data processing(e.g.,data mining) and data storage services for developers are supported by either a database and/or a distributed?le system.These latter elements can(since it is not mandatory to have an IaaS cloud to build a PaaS),in turn,be deployed inside one or more VMs on top of the capabilities enabled by the IaaS layer.

In educational environments,both IaaS and PaaS can be considered as supporting technologies for education.From the features described,a several-fold increase in ef?ciency (improved access to widespread heterogeneous resources with reduced costs[1])can be expected,helping educators and students to reach the intended educational results faster.In ad-dition,another fact is also attracting educators’focus to cloud technologies:Several major IaaS vendors,such as Rightscale and Amazon,have recently presented their“education pro-grams”to ease educational access to cloud infrastructures.4 Other players have also offered mail,collaboration,and more software in the cloud,such as Microsoft’s Live@Edu.5

Given this is something of a Cambrian era for a widespread application of the cloud to education,the main goal of this study is to ascertain what type of cloud is more appropriate to reach the desired educational objectives.To accomplish that mission, students in an advanced course in computer science have been manually setting up networks and nodes and coding the required monitoring and routing algorithms for creating an overlay net-work that they used for assignments involving distributed soft-ware applications.This paper investigates whether the use of IaaS or PaaS(as infrastructure for the assignment)increases or decreases the performance of the students and how they perceive the use of IaaS and PaaS.That is,do students using IaaS or PaaS ?nd it to be easier or harder than manually setting up the needed infrastructure to test the probing and routing algorithms(main 47d8b26f84693daef5ef73d28/products/plans-pricing/cloud-computing-for-education.php or 7d8b26f84693daef5ef73d28/education/.Last viewed December 2010.

57d8b26f84693daef5ef73d28/7d8b26f84693daef5ef73d28st viewed December2010.focus of each assignment),and do they have a clear preference for one technology over the other?The perception of the pro-fessors was also taken into account.

The remainder of this paper is structured as follows. Section II presents related work in previous distributed systems and paradigms that tried to improve the ef?ciency in education. Having precedent technologies pros and cons clear,Section III introduces the educational context,the learning objectives of the course,and the cloud technologies to be used to achieve the objectives.Then,the obtained results are presented in Section IV.This quantitative evaluation is further explained by the inclusion of qualitative information in Section V.Finally,a wrap-up of the obtained results presented in Section VI.

II.R ELATED W ORK

Among available IT,several distributed technologies have been employed to support education by increasing the range of available resources at a reduced cost.

SOAs advocate for the usage of open and standard-based in-terfaces to facilitate the reutilization and integration of several educational tools[2],[3],[8],[9].Examples of service-based educational environments include LearnServe[10].This system allows pupils to search and select learning contents that are later presented sequentially.

Other service-oriented approaches introduced the usage of grid services for enabling widespread access to computational and hardware resources[4]–[6],[11],[12].As discussed in[7], grids and clouds are different paradigms whose differences and similarities render the distinction fuzzy.For the sake of brevity, the differences can be summarized by recalling the four main elements offered by a cloud:on-demand provision,SLAs,scal-ability,and a pay-per-use billing model.The interested reader is referred to[7]for a more detailed comparison.Gimolus was an environmental learning grid-enabled tool consisting of a topo-logical database joined to a visualization tool that let pupils study the effects of several phenomena on the ground[11].Sim-ilarly,COV ASE let users’avatars interact in a virtual environ-ment to learn?uid dynamics and?nite state analysis[5].These were shown in real time,and needed high computational power to reach optimum performance and usability.In terms of hard-ware resources,Isilab built a lab consisting of remotely con-trolled electronic equipment such as a signal generator,an oscil-

592IEEE TRANSACTIONS ON EDUCATION,VOL.54,NO.4,NOVEMBER 2011

TABLE I

E DUCATIONAL E V ALUATION O

F THE A V AILABLE C LOUDS A IMED AT E DUCATIONAL E NVIRONMENTS .N OTE :S AA S IS N OT C OVERED IN T HIS S TUDY ,

B UT IT IS I NCLUDED H ERE FOR C

OMPLETENESS

loscope,and a patch panel relying on proprietary protocols and presenting a proof of concept [12].In addition to access to hard-ware and computing power,Gridcole offered mechanisms to guide learners through the realization of collaborative learning situations according to sequences of activities as speci?ed in “learning scripts”written by lecturers [6].

Although they all allowed for the implementation of un-precedented learning situations,these service-oriented systems (including grid-service based ones)relied on statically provi-sioned resources:Computing power and the required hardware had to be present before they could be used.They also had to be federated (under an administrative procedure that shares these resources with a selected group of organizations also sharing their own resources).This need for shared on-site resources resulted in higher costs (reduced ef?ciency)for permanently-on resources.Signi?cantly,service-oriented approaches still repre-sented an important impact of the supporting technologies:the selection and aggregation (integration)of often incompatible,ready-to-use services.Any deviation from this selection/ag-gregation pattern typically requires the action of a technical administrator [6],[13].

Cloud computing has not remained unaware of education’s importance,and it promises users that they can employ their de-sired applications and hardware resources without the necessity of,for instance,purchasing the software and having the frustra-tion of installing it or dealing with differences among versions,licensing,etc.A several-fold increase in ef?ciency (persity of resources,ease of use,and reduced cost)can thus be expected,thereby helping educators and students to reach the intended re-sults faster.

There are already some proposals to apply cloud technolo-gies in educational environments;see Table I.Some relevant efforts to use IaaS cloud technologies in education include StarHPC [14].StarHPC prepackaged a VM image used by students,the scripts used by an administrator,and a virtual image of the Amazon Elastic Computing Cloud (EC2)machine used to build the cluster shared by the class.At the PaaS level,more open and testable experimental platforms are needed in a lab-level with PCs.Tian et al.[15]presented a higher abstraction-level framework for managing PaaS in a virtual cloud computing lab.Yin et al.built a GAE-based course sharing platform [16],but did not perform an evaluation of the usefulness of the cloud for learning purposes.Also,Seattle offered a community-based effort to set up a platform envi-ronment [online APIs offering capabilities for managing ?les,network resources,threads (called timers and locking APIs),and miscellaneous (other online libraries for doing exits,etc.)]to help to deliver a cloud in the educational environment that promoted resource usage [17].Other authors proposed a layered architecture for an e-Learning cloud [18],focused on delivering SaaS.Similarly,Hu and Zhang presented a case study of a

mechanical engineering course redesign where learning effec-tiveness is compared to its traditional and online-only course delivery [19].As mentioned in Section I,SaaS aims at offering online services to end-users,so in this regard it is similar to any general online learning tool.As another type of cloud,SaaS clouds still bene?t from the SLAs,dynamic provision,automated scaling,and pay-per-use models that characterize the cloud capabilities mentioned in the Introduction,which is an important difference to traditional online learning platforms.All this interest in clouds is justi?ed by the perceived advan-tages they bring.However,this increasing number of tools and architectures for IaaS/PaaS supported education has not been accompanied by a real assessment on the effectiveness of these approaches (Table I).More speci?cally,column 3of Table I,reveals that an inpidual and comparative evaluation is still missing for IaaS and PaaS in order to ascertain their real po-tential in some scenarios.

Of all the cloud-related works,only Hu et al.[19]performed a comparison on the ef?ciency of online models versus tradi-tional models.This evaluation was not focused on the impact of supporting technologies or the perceived ease of use and ac-celeration of the learning process.Furthermore,the appropriate level of abstraction (i.e.,IaaS or PaaS)that should be delivered to students to enable them to focus on the course topics has not been analyzed by previous works.That is,previous reports do not present an exhaustive evaluation of the advantages of using different type of clouds and the educational outcome obtained (average quali?cation,perceived ease of usage,time to obtain the desired results,etc.).

This paper presents data on how the utilization of clouds can be useful for achieving not only resource-usage/cost optimiza-tion,but also for enhancing students’results.Furthermore,the most appropriate abstraction level at the cloud stack (IaaS or PaaS)is analyzed.

III.C OURSE B ACKGROUND

A.Course Description

The course on overlay networks described here.“Infraestruc-tura de Redes”is taught in the ?fth year of study in the Computer Engineering degree at Universidad Rey Juan Carlos (URJC),Móstoles,Spain.This degree requires the completion of ap-proximately 35–40credits (European Credit Transfer Systems,ECTS)per semester during a ?ve-year period.The course intro-duces overlay networks and peer-to-peer systems and discusses their general properties and applications,drawing its material from all these content areas.The range of topics dealt with is very varied,ranging from the optimal placement of a node in the overlay,how to monitor the servers and networks connecting them,and how to discover,access,and distribute information in large-scale graphs.

V AQUERO:EduCloud:PaaS VERSUS IaaS CLOUD USAGE FOR ADV ANCED COMPUTER SCIENCE COURSE593

This course is closely related to material taught in several computer science and engineering curricula:System Architec-ture and Organization and Algorithms and Complexity(ACM CS2008curriculum);in Algorithms,Computer Systems Engi-neering or Computer Networks(ACM’s CE2004curriculum); Algorithms and Data Structures(ACM’s IS2002curriculum); Networking,Platform Technologies,(ACM’s IT2008cur-riculum);and Net-centric Computing and Algorithms and complexity in K12’s education curriculum.These are very im-portant content areas in their respective curricula and,as shown below,can be bene?ted from the usage of cloud technologies. Clouds are relevant to several subjects in these curriculum such as networking,system administration,economics,algorithms, platforms,etc.More speci?cally,the course covers the fol-lowing topics:

?introduction to complexity theory;

?structured and unstructured networks,power-law distributions;

?currently deployed peer-to-peer systems and how they work;

?examples of today’s overlay networks and differences with their antecedents;

?distributed Hash tables as a basis for structured peer-to-peer systems;

?overlay networks design considerations and general ar-chitecture from internetworking to resource management: monitoring and routing algorithms;

?content delivery networks,their hierarchy,and their perfor-mance evaluation;

?performance issues,monitoring performance,networking tricks,legal aspects,and privacy issues;

?content distribution algorithms;

?peer/node discovery algorithms(central versus distributed systems).

This syllabus is accompanied by the realization of several lab assignments and a?nal practical session whose purpose is for students to connect the concepts they met in the previous labs. This last practical is explained in detail in Section IV.

B.Evaluation Setup

Eighty-four students carried out two lab experiments using three different setups.The?rst(IaaS)uses an IaaS cloud for the provision of VMs,the second(PaaS)uses a PaaS cloud to host student software,and the third(control)is similar to previous years’settings where students were in charge of the whole setup. The?rst group consisted of a series of54computer science undergraduate students at Telefónica Labs,Madrid,Spain,be-tween May and July2010,as well as10junior programmers (former undergraduate students)with less than two months of working experience.Since most of the undergraduate students had little practical experience,the presence of both graduate stu-dents and junior programmers with higher expertise helped to analyze whether the familiarity with programming languages, server con?guration,and the like made any difference.This also helped to improve the statistical power of the tests.All students underwent an inpidual evaluation of their expertise.6

6Indeed,in this evaluation,three junior programmers got low rates and were classi?ed as elementary

students.Fig.2.Experiences employed for comparison of the different educational sit-uations enabled by cloud technologies.(a)Control.(b)IaaS.(c)PaaS. Later,a second group of20?fth-year undergraduate computer science students with college-level programming skills at Universidad Rey Juan Carlos joined the study,re-peating the same experimental conditions between July and September2010.The experiments were directed by four ad-junct professors working at Telefónica Labs and Universidad Rey Juan Carlos.

The aim of the lab experiment was to build an overlay net-work where nodes communicate by using conventional HTTP protocols,so that there is a single central register by which all nodes are known(this frees students from having to spend their time designing and implementing node discovery algorithms). Every node has a list of the other nodes and the distance at which they lie.This distance is obtained by aggregating different net-work measurements(every node has a probe attached to it to get this data;students design the way the nodes and the network are probed).Data are sent through the fully connected network via the less expensive path according to probe measurements (routing algorithms are an essential design decision).To build such an application,three different scenarios are envisioned.?Designing a simple servlet by relying on traditional programming frameworks such as Eclipse and Apache Tomcat.7Students had to install the operating system,set up networking,set up certi?cates with a local certi?cation authority to access lab machines via SSH,and after this, con?gure Tomcat and start developing a simple servlet serving as central registry for overlay nodes(which are servlets themselves).This is the setting denoted as“con-trol”since it allows traditional scenarios to be compared to cloud-based ones.See Fig.2(a).

?Deploying of a worldwide spread network of?ve Amazon EC2Linux nodes so that students just have to deploy the re-quired VMs(a simple click)when needed,install Tomcat, and start developing,as shown in Fig.2(b).This is the set-ting denoted as“IaaS.”

?Finally,employing[Fig.2(c)]GAE to build the application (students just have to install an Eclipse plugin to interact transparently with GAE and start developing).This is the setting denoted as“PaaS.”

IV.R ESULTS O BTAINED:E V ALUATION

A.Evaluation Process Design

As with previous studies[6],[20],[21],the evaluation process is pided into three different stages.

1)Categorization of the concepts under study.

77d8b26f84693daef5ef73d28/.Last viewed December2010.

594IEEE TRANSACTIONS ON EDUCATION,VOL.54,NO.4,NOVEMBER 2011

TABLE II

I NITIAL S CALE I TEMS FOR P ERCEIVED U SEFULNESS AND E ASE OF U SE .E VERY I TEM IN THE T ABLE W AS R ATED F ROM 1TO 4,M EANING D ISAGREE ,

P ARTIALLY D ISAGREE ,P ARTIALLY A GREE ,AND T OTALLY A GREE ,R

ESPECTIVELY

Fig.3.Student grouping for results analysis.The numbers in parentheses in-dicate the actual number of students assigned to each group.

2)Data collection ,in this case including perse data sources such as pretest surveys in which students were asked their degree of experience and classi?ed into advanced (meaning more experienced)and elementary (or least experienced)students.The surveys were designed to cover all the categories identi?ed in the ?rst stage of the evaluation.Every item in these categories was rated from 1to 4,meaning disagree,partially disagree,partially agree,and totally agree,respectively (Table II).Also,data regarding the order of introduction to IaaS/PaaS,the time devoted to each task,and the level of knowledge acquired were collected.The second data source was discussion groups.An inpidual meeting was held with students and professors to discuss key elements in the surveys and to deepen in the evaluation these performed

3)Data analysis .The data gathered were analyzed by com-bining a quantitative and qualitative approach,combining experimental data,surveys,and inpidual interviews [21].Section IV-A1introduces some of the relevant elements for the surveys and data the analysis.Some results of the quantitative analysis are better understood in conjunction with explanations and reasons given by professors and students.In other words,this qualitative analysis supports and explains the statistical ?ndings expressed before.This combination is shown in Section V.

1)Employed Categories:A ?rst category is related to the previous experience of the students in related tasks.As men-tioned,an initial survey is employed for categorizing

students

Fig.4.Results of the data analysis process performed to the systems by stu-dents and professors.The results include crossover comparisons for different groups and considering survey answers as in Table II and quantitative data of time to completion,time devoted to each phase,and percent of success.

into two classes:advanced (more experienced)and elementary (less experienced)(see Fig.3).The survey analyzed questions such as their previous experience in system management,and it was performed by formulating four simple questions to the students trying to get a hint of their knowledge on Java Servlet programming and hardware con?guration (including security,networking,etc.).

Having this initial segmentation,crossover experiments 8were selected.Six groups were built,each one corresponding to a sequence of two of the three possible con?guration set-tings (control,IaaS and PaaS;see Section III).Students were randomly assigned to one of those six groups.The goal is to be able to perform fair comparisons between all possible settings to describe the “goodness”of every employed system (as shown in Fig.3).In this inpidual comparison,the terms employed for students to rate different approaches are based on the work by Davis [22].More speci?cally,this value is analyzed from a student point of view (perceived ease of use and the perceived usefulness,as shown in Table II)and from the lecturer’s perspective (time to complete the task,average rating and percentage of students completing the task in time).An estimation of the time devoted to each stage by students was also included in the survey.

2)Data Analysis:After completing the data collection stage (see Fig.4),a per-group (per-sequence)evaluation of the results was obtained after students were placed in their ?rst set (control,IaaS or PaaS)in the sequence.Then,every student serves as his/her own control for the comparison by performing an anal-ysis of variance test (anova)between the two settings with which he/she has worked.The effects of the order in the sequence of settings are also studied.Professors are given a ?nal survey on pedagogic questions.

87d8b26f84693daef5ef73d28/wiki/Crossover_7d8b26f84693daef5ef73d28st viewed December 2010.

V AQUERO:EduCloud:PaaS VERSUS IaaS CLOUD USAGE FOR ADV ANCED COMPUTER SCIENCE COURSE

595

Fig.5.Student survey to assess their initial experience.The numbers in parentheses indicate the actual number of answers to the

question.

Fig.6.Independent comparison of the different scenarios.indicates p <0:05as compared to control for a t -test for independent samples with n =28.

B.Evaluation Results

As mentioned,for the initial survey’s results,students were evenly classi?ed into advanced (42students who had gener-ally used the required technologies many times)and elemen-tary (42who had used most of the technologies a few times or never)students in order to ?nd differences in the results due to their background level (see Fig.5).

Usefulness and ease of use were analyzed by calculating the average rating of every student of all the items in Table II.First,the answers from all the students going ?rst to control versus ?rst to IaaS and versus ?rst to a PaaS were analyzed.Results are

shown in Fig.6.Control-?rst students (group

size

)rated this approach’s usefulness and ease of use as

1.770.03and

2.090.01,respectively.Students ?rst submitted to an IaaS en-

vironment

rated usefulness as

2.160.01and ease of use as

2.430.0

3.Finally,students under a PaaS environment

?rst

gave

1.750.03to usefulness and

2.170.02to ease of use.Students rated IaaS better than control or PaaS ,although the differences in the initial evaluation of the different scenarios were minor with regard to their usefulness and ease of use.These results remained unaltered in spite of the initial experience of the students.

However,an “apple to apple”comparison called for the consideration of every inpidual student and his/her perceived variation in usefulness and ease of use when comparing two different situations.Results of such comparisons are shown in Fig.7.When elementary students were ?rst put in a con-trol environment,they rated the usability of the approach at

1.55

0.05.When this same set of students was asked

to rate again after producing the same code in an IaaS scenario,

this was signi?cantly increased to

2.86

0.04

(

;)[see left-hand side of Fig.7(a)].Doing the opposite [see Fig.7(b)]experiment revealed similar features,and students rated the option of IaaS signi?cantly higher than the control in both usefulness and ease of use (see Fig.7).

Fig.7(c)and (d)reveals the differences in the perceived use-fulness and ease of use by students using control conditions and a PaaS cloud for developing the desired overlay.As can be ob-served,students consistently rated PaaS higher than the control no matter the order of the sequence .It should be pointed out that students tend to rate the control option lower after having used the PaaS tools for developing their code (as also happened whenever students ?rst used IaaS and were then transferred to a more conventional control scenario).Finally,Fig.7(e)and (f)di-rectly compares students ?rst exposed to IaaS and then to PaaS,and vice versa.Students found PaaS to be more useful and easier to use than IaaS no matter their initial level of expertise in the tasks at hand.Again,IaaS was rated lower after having used a PaaS environment.

A general observation from Fig.7is that,in all sequences,there were no relevant differences between the perceptions of advanced (experienced)and elementary (less experienced)stu-dents.Also notable is that 85%of the students using PaaS rated it as very easy to use,learn,and remember (4out of 4rating).Similarly,80%of the students rated this to be most useful (4over 4rating)for saving time and helping to focus on the course topics (algorithms and overlay layout).This unanimity was not achieved for IaaS over control,in spite of the signi?cantly im-proved student perception.

With these data at hand,the discussion meetings were started.All students were prompted to quantify the “cloud-induced time reduction”depending on the group they were in,with the fol-lowing results:

1)Control versus IaaS :

15.0 1.2h vs

11.30.8

h

;2)Control versus PaaS :

13.8 1.1h vs

7.70.6

h ;3)IaaS versus PaaS :

10.6 1.1h vs

6.8 1.4

h .A second question addressed in these brie?ngs was the es-timated time devoted to the tasks required for completing the process;see Table III.The values obtained do not exactly add up

596IEEE TRANSACTIONS ON EDUCATION,VOL.54,NO.4,NOVEMBER

2011

Fig.7.Crossover comparison of the combination of two consecutive different scenarios.indicates p <0:05for n =7.The numbers in the bars indicate the order in which students were submitted to the different learning scenarios.(a),(b)Students under a control and an IaaS scenario,and vice versa,respec-tively.(c),(d)Students under a control and a PaaS scenario,and vice versa,respectively.(e),(f)Students under an IaaS and a PaaS scenario,and vice versa,respectively.

TABLE III

T IME D EVOTED TO A LL THE T ASKS FOR THE D IFFERENT S CENARIOS .T IME IS

S HOWN IN H OURS 6S TANDARD E RROR OF THE M EAN FOR n =

28

to those reported in the list above,but they remain within reason-ably similar limits.During the brie?ngs in discussion groups,most students agreed that cloud computing techniques made their lives easier and helped them to focus on the “overlay”problem “rather than dealing with stuff for other subjects.”After the brie?ngs,a test was prepared by the professors to check the knowledge acquired by students regarding overlay probing and algorithms (typical questions included the theoretical design of a content distribution system taking into account node’s loca-tion and the most appropriate techniques for probing and routing content across the overlay).The number of passing students (considering students at Universidad Rey Juan Carlos only)in-creased by 30%as compared to the previous year’s data.These data are not directly comparable since the students in this course are exposed to two different setups (in the previous year,stu-dents underwent “control”conditions only).

The four professors were asked to rate the following sen-tences from 1–4depending on their degree of agreement

(mean standard error in the mean;ratings were between 1and 4as shown):

1)IaaS helps knowledge acquisition :

1.5

0.14

;2)IaaS eases evaluation :

1.25

0.13;3)IaaS facilitates students work :

3.25

0.13;4)IaaS helps students’focus :

3.0

0;5)IaaS helps attain objective :

3.25

0.24;6)PaaS facilitates knowledge acquisition :

1.5

0.14

;

7)PaaS helps evaluation :

1.25

0.13;8)PaaS helps students work :

3.5

0.14;9)PaaS facilitates students’focus :

3.75

0.13;10)PaaS helps target accomplishment :

4.0

0.

As can be observed,both cloud approaches increase the per-ception by professors that cloud computing helps attain the ed-ucational objective and helps students to focus on the problem at hand,although neither help in evaluating students.

Rather,brie?ngs with professors revealed it took them more time to prepare classes for cloud-related lab

assignments,30%increase in the preparation time.Professors had to learn how to manage and set up some VMs in Amazon (simple Web interface for VM deployment,which also implies some con?guration steps).These steps took only a little time for them to learn (the average estimated time to control this environment

was

h,)with little perceived complexity (“the Web interface certainly helped to reduce errors and to set up the environment”or “having VMs deployed in three minutes is a de?nite help that allows one to forget about low level stuff”).The increase in preparation time is principally derived from the newness of the technologies since appropriate images had to be generated and uploaded to Amazon’s simple storage service (S3)for later use.Amazon EC2tools had to be included in the images to allow for persistence of the running images (otherwise,all the changes are lost upon reboot of the machine).Learning these APIs and creating and updating the images was deemed as the most time-consuming task for lab assignment preparation.For GAE,two of the professors involved had less background on software programming.Even so,they found no problems in integrating GAE’s Eclipse plugin and using it for implementing the probing and routing algorithm.

V .D ISCUSSION

The changes the cloud is already bringing to how students and academics (and administrators)actually work can be observed in a variety of recent software [14]–[17],[19].These present

V AQUERO:EduCloud:PaaS VERSUS IaaS CLOUD USAGE FOR ADV ANCED COMPUTER SCIENCE COURSE597

tools,frameworks,and software packages are offered as a ser-vice in educational scenarios.However,a detailed evaluation of the educational bene?ts of cloud computing for education was still missing.The evaluation of infrastructure and platform clouds,either alone or as compared to one another,yields a hint of what the most appropriate cloud type could be in an advanced course on overlay networks,in the assignments for which stu-dents need to deploy distributed infrastructure and decide/im-plement the most appropriate probing and routing algorithms. An initial comparison of students subject to the?rst scenario in their sequence(either control,IaaS,or PaaS)revealed minor differences between these technologies.This is explained by the fact that students tend to complain about the platform they are using,so they were put in a more debatable scenario and forced to compare two different approaches.Thus,a crossover study was employed with all84students,randomly gathered in seven-member groups(sequences).An important point with these se-quences of conditions has to do with the selected order since this may affect the outcome.For instance,a user?rst randomized to a PaaS platform may have a better opinion than one?rst ran-domized to a control group.To analyze this bias,a comparison of reverse-order sequences(e.g.,results when students?rst used IaaS and then PaaS versus results when using?rst PaaS and then IaaS)was performed,showing minimal differences in order. Of course,a“learning”effect is inherently there in the evalu-ation.This effect has to do with the knowledge that students carry over to the next sequence(e.g.,knowledge on probing and routing—see Fig.2—will be useful at all stages,while knowl-edge on hardware con?guration will not prove useful for an IaaS or PaaS scenario).This was taken into account when analyzing PaaS to IaaS,PaaS to control,and IaaS to control groups.To minimize this effect,a whole month“washout”period was al-lowed in which students did not work on related topics,and the algorithms(for appropriately probing,for routing informa-tion,for content distribution,etc.)used in each experiment were changed.This minimization is shown by the fact that no differ-ences in the?nal evaluation were found and the time for de-veloping probing and routing(see Table III)is not signi?cantly different for all three scenarios considered.

The principal observations to be made from this analysis follow.

?Overall,students and professors agreed that cloud tech-niques helped to maintain students’focus.Professors indi-cated their perception that cloud technologies saved a lot of time for developing the course contents.Of course,using PaaS clouds for courses on(for example)operating system administration may soon prove to be a poorly conceived idea since these hide the underlying heterogeneity of the operating system employed and the software tools used.

Clouds should not be applied when the cloud could isolate students from the technologies/concepts they are supposed to learn.

?In general,students valued PaaS for its ease of learning, remembering,and use and for the time it saved.More vari-ability was found in the?nal quality of the work(no dif-ferences were found that were dependent on the previous background of the students).Regarding this overall quality, it should be pointed out that in spite of saving a lot of time,cloud technologies did not prompt students to devote the time saved to overlay probing and routing algorithms.

These data obtained by student ratings are in full agreement with the lack of improvement in the grades obtained in?nal tests(e.g.,using PaaS did not lead to better marks than did using IaaS).They are also in agreement with the experience of the professors,who did not consider any cloud abstrac-tion level to improve the results obtained or the acquisi-tion of new knowledge by students.Of course,technology “per se”is not enough to help to motivate students further so that they devote more hours to the routing and probing problem and thus increase their?nal grade.This should be considered as a by-product of the study,not as a main target.

?A signi?cant result of including cloud in this course was the marked increase in the percentage of passing students as compared to the previous year.Both students and profes-sors,interviewed after the tests,agree that cloud technolo-gies helped them to be more focused on the task at hand,re-sulting in a higher percentage of passing students.Quoting

a student,“I felt I could focus on designing the algorithms

and testing them in a real environment,rather than either wasting a lifetime on platform tests or performing mere simulations,”while another one mentioned that“I liked GAE(PaaS)the most,since it took out most of the un-necessary underlying complexity,while letting one focus on the algorithmic design.”On the other hand,professors realize that“I can devote more time to explaining mathe-matical tricks and algorithms,instead of wasting time in setting up the environment,”or“Although I was reluctant at the beginning,the experiment was worth it and students seemed satis?ed.”Of course,comparing these data to those of the previous year only cannot be considered as statisti-cally signi?cant,so readers should take this cautiously.?Another positive issue that arose in the discussion with stu-dents after the tests had to do with the ubiquitous access and worldwide infrastructure that can be set up for them, rather than their relying on local resources.This fact has also been pointed out by others as“the key mobile role for education”[1].There,the unbundling by institutions(sepa-rate courses from academic programs and even course con-tents)and custom use of resources for professors and stu-dents to access the content were highly recommended[1].?An inconvenience highlighted by professors was that the cloud itself did not only fail to help in boosting evalua-tion,but also it increased the time devoted to lab assign-ment preparation.The fact that they were doing this for the?rst time and did not have an in-depth knowledge of the technological environment was pointed out by most of the professors.These factors will diminish as technologies get established in the course curriculum and should not be

a problem for future offerings of the course.Students,in

turn,complained about how it is hard for them to get used to this cloud philosophy when they are introduced to it in the?rst lab session.A“penalty effect”was also noticed by which control rates are lower after having used IaaS or PaaS,and IaaS is poorly rated after having used PaaS.This observation was con?rmed by some students claiming that “we felt like dealing with unneeded stuff when turned back down to an IaaS/PaaS option.”

Examining curricula above the cloud could potentially bene?t subjects such as networking,system administration,algorithms,

598IEEE TRANSACTIONS ON EDUCATION,VOL.54,NO.4,NOVEMBER2011

and platforms,depending on the required abstraction level and where the learning elements should be located.For instance,if a lecturer wants lab assignments to focus on high-level tasks, such as software development,algorithms,or the like,a PaaS cloud hides the underlying layers’complexity.On the contrary, if the aim is to train a system administrator,a more integrative approach including all layers(from hardware to software)may be required(this is where the cloud does not come in handy in computer science and engineering).

While it seems ideal for selecting and focusing on the tasks where students should focus their attention,removing unneeded complexity,IaaS or PaaS clouds are more dif?cult to extrapo-late to other subjects beyond computer science and engineering (PaaS clouds still require students to code parts of the applica-tion,although part of the dif?cult material,such as data persis-tence or security,is left out and automatically provided by the platform).They can very well serve as a basement technology for delivering SaaS capabilities to students in other areas(see SaaS references in Table I).

VI.C ONCLUSION

This paper presents the?rst evaluation of the actual bene-?ts of cloud computing for a course on overlay networks with a focus on probing and routing algorithms.The results reveal that the introduction of cloud technologies is appropriate to main-tain students’focus and save time on non-course-related tasks (reducing the problem inherent to the use of technological sup-port for education),while it does not help to motivate students and does not actually improve the average results.PaaS clouds are almost unanimously considered as easy to learn and use and may become a prevalent option for high-level courses.In turn, IaaS clouds will serve for medium abstract-level courses(such as those on operating systems,containers,or software con?g-urations),while the traditional approach where students are in charge of everything should be kept for advanced system man-agement courses only.

A CKNOWLEDGMENT

The author would like to thank students and professors at URJC and Telefónica Labs for their kindest help and time.The author would also like to thank Dr.L.Rodero-Merino for proof-reading the paper and providing suggestions and comments.

R EFERENCES

[1]B.Alexander,A.Attis,Y.Benkler,A.Cooley,P.Courant,J.Davis,

G.Davis,L.Faulkner,I.Fuchs,P.Gandel,C.Geith,P.Goldstein,

B.Hawkins,R.Katz,7d8b26f84693daef5ef73d28ne,

C.Lynch,M.Marlino,P.Mclean,

D.

Oblinger,L.O’Brien,M.Read,T.Sumner,J.Unsworth,B.Wheeler,

K.Woolsey,and R.Yanoski,The Tower and The Cloud:Higher Edu-

cation in the Age of Cloud Computing,R.N.Katz,Ed.Boulder,CO:

EDUCASE,2008.

[2]D.Dagger,A.O’Connor,7d8b26f84693daef5ef73d28wless,E.Walsh,and V.P.Wade,“Ser-

vice-oriented e-learning platforms:From monolithic systems to?ex-

ible services,”IEEE Internet Comput.,vol.11,no.3,pp.28–35,May/

Jun.2007.

[3]C.Fang and L.C.Sing,“Collaborative learning using service-oriented

architecture:A framework design,”Knowledge-Based Syst.,vol.22,

no.4,pp.271–274,2009.

[4]L.M.Vaquero-González,D.Hernández-Leo,F.Simmross-Watten-

berg,M.L.Bote-Lorenzo,J.I.Asensio-Pérez,Y.A.Dimitriadis,

E.Gómez-Sánchez,and G.Vega-Gorgojo,“The opportunity of grid

services for CSCL-application development,”in Proc.Euromicro

Conf.Parallel,Distrib.,Network-Based Process.,Los Alamitos,CA,

2005,pp.4–11.

[5]P.J.Beers,P.A.Kirschner,H.P.A.Boshuizen,and W.H.Gijselaers,

“Coercing knowledge construction in collaborative learning environ-

ments,”in Proc.CSCL,2005,pp.8–17.

[6]M.L.Bote-Lorenzo,E.Gómez-Sánchez,G.Vega-Gorgojo,Y.A.Dim-

itriadis,J.I.Asensio-Pérez,and I.M.Jorrín-Abellán,“Gridcole:A tai-

lorable grid service based system that supports scripted collaborative

learning,”7d8b26f84693daef5ef73d28c.,vol.51,no.1,pp.155–172,2008.

[7]L.M.Vaquero,L.Rodero-Merino,J.Caceres,and M.Lindner,“A

break in the clouds:Towards a cloud de?nition,”7d8b26f84693daef5ef73d28mun.

Rev.,vol.39,no.1,pp.50–55,2009.

[8]Z.Xu,Z.Yin,and A.Saddik,“A Web services oriented framework for

dynamic e-learning systems,”in Proc.IEEE CCECE,May2003,vol.

2,pp.943–946,vol.2.

[9]K.Friesen and N.Mazloumi,“Integration of learning management sys-

tems and Web applications using Web services,”Int.J.Adv.Technol.

Learn.,vol.1,no.1,pp.16–24,2004.

[10]G.V ossen and P.Westerkamp,“E-learning as a web service,”in Proc.

Int.Database Eng.Appl.Symp.,2003,p.242.

[11]S.Wesner,K.Wulf,and M.Müller,“How grid could improve

e-learning in the environmental science domain,”in Proc.1st

LeGE-WG Int.Workshop Educ.Models Grid Based Services,2002,

pp.3–4.

[12]A.Bagnasco,A.Poggi,and A.Scapolla,“Computational grids and

online laboratories,”in Proc.1st Int.Conf.Adv.Technol.Enhanced

Learn.,2005,pp.14–19.

[13]C.Alario-Hoyos,J.I.Asensio-Pérez,M.L.Bote-Lorenzo,E.Gómez-

Sánchez,G.Vega-Gorgojo,and A.Ruiz-Calleja,“Integration of ex-

ternal tools in virtual learning environments:Main design issues and

alternatives,”in Proc.10th IEEE Int.Conf.Adv.Learn.Technol.,Wash-

ington,DC,2010,pp.384–388.

[14]C.Ivica,J.Riley,and C.Shubert,“Starhpc:Teaching parallel program-

ming within elastic compute cloud,”in Proc.31st ITI,Jun.2009,pp.

353–356.

[15]W.Tian,S.Su,and G.Lu,“A framework for implementing and man-

aging platform as a service in a virtual cloud computing lab,”in Proc.

2nd ETCS,Mar.2010,vol.2,pp.273–276.

[16]H.Yin,J.Han,J.Liu,and X.Hongyun,“Development and research

of multimedia courseware sharing platform based on GAE,”in Proc.

EDT,Apr.2010,vol.2,pp.289–292.

[17]J.Cappos,I.Beschastnikh,A.Krishnamurthy,and T.Anderson,

“Seattle:A platform for educational cloud computing,”in Proc.40th

ACM SIGCSE,New York,2009,pp.111–115.

[18]Z.Yang and Z.Zhu,“Construction of OSS-based e-learning cloud in

China,”in Proc.2nd ICETC,Jun.2010,vol.2,pp.V2-398–V2-401.

[19]Z.Hu and S.Zhang,“Blended/hybrid course design in active learning

cloud at South Dakota State University,”in Proc.2nd ICETC,Jun.

2010,vol.1,pp.V1-63–V1-67.

[20]A.Martinez-Mones, E.Gomez-Sanchez,Y.Dimitriadis,I.

Jorrin-Abellan, B.Rubia-Avi,and G.Vega-Gorgojo,“Multiple

case studies to enhance project-based learning in a computer archi-

tecture course,”IEEE 7d8b26f84693daef5ef73d28c.,vol.48,no.3,pp.482–489,Aug.

2005.

[21]A.Martínez,Y.Dimitriadis,B.Rubia,and E.Gómez,“Combining

qualitative evaluation and social network analysis for the study of class-

room social interactions,”7d8b26f84693daef5ef73d28c.,vol.41,pp.353–368,2003.

[22]F.D.Davis,“Perceived usefulness,perceived ease of use,and user

acceptance of information technology,”MIS Quart.,vol.13,no.3,pp.

319–339,Sep.1989.

Luis M.Vaquero(M’04)received the B.Sc.degree in electronics,M.Sc.degree in electrical engineering,and Ph.D.degree in information technologies from Universidad de Valladolid,Valladolid,Spain,in2001,2003,and2010,respec-tively,and the M.Sc.degree in pharmacology and Ph.D.degree in medicine from Universidad Complutense de Madrid,Madrid,Spain,in2006and2009, respectively.

After graduating,he was a Research Associate with several U.S.centers and a Researcher and IPR Manager for Telefónica,Madrid,Spain.He is currently a Part-Time Assistant Professor with Universidad Rey Juan Carlos,Móstoles, Spain.He recently joined Hewlett-Packard Labs,Bristol,U.K.His research in-terests are in distributed systems and their application to other?elds,such as social,educational,or biomedical sciences.

本文来源：https://www.bwwdw.com/article/h7vl.html