IDEAS: Instructional Design for Elearning ApproacheS

Reviewing Synchronous Learning Objects

In response to Christy's comments regarding the use of LORI in sychronous environments.

The creators of the LORI describe using a convergent participation evaluation model in conjunction with the rating instrument.  Please see;
Nesbit, J., Belfer, K., & Vargo, J. (2002). A convergent participant model for evaluation of learning ob-jects. Canadian Journal of Learning and Technology, 28 (3). Retrieved  from http://www.cjlt.ca/content/vol28.3/nesbit_etal.html

I believe that both could be applicable for digital objects available synchronously as many of the criteria in the LORI would be relevant; feedback adaptation; learning goal alignment, content quality, presentation design.

You might want to adapt the criteria in the LORI to address some of the dynamics that are present during real-time activities, i.e. ability to guage understanding and respond to the needs of learners.

Learner motivations in a business and academic environment may not be the same, but there needs to be a formal reward system in place to recognize those who take part in the review process.  Recognition also makes the activity more credible, because in effect the organization/ insitution is reinforcing the idea that this activity is a valuable undertaking.  In academia, the idea behind engaging in a rigorous evaluation of the quality of learning materials is to promote it as a scholarly activity similar to articles being peer reviewed for a journal.  Having someone use your LO would be equivalent to a citation in an article which may indicate that your resource is of some value in that particular discipline. One way of recognizing people who engage in these activities in a corporate environment would be to provide them with the training and time to develop/ evaluate these materials.

Learning Object Articles

A number of articles relating to Learning Objects have just been published online in the Interdisciplinary Journal of Knowledge and Learning Objects, including:

Interoperability and Learning Objects: An Overview of E-Learning Standardization, by Norm Friesen.

Interactive QuickTime: Developing and Evaluating Multimedia Learning Objects to Enhance Both Face-To-Face and Distance E-Learning Environments, by Thomas Cochrane.

Learning Objects: Using Language Structures to Understand the Transition from Affordance Systems to Intelligent Systems, by Jacques du Plessis.

A Study of the Design and Evaluation of a Learning Object and Implications for Content Development, by Ferdinand Krauss & Mohamed Ally.

The links to the articles have been updated and all work now. Thanks to Keith for letting me know. All articles are in Adobe Acrobat (PDF) format.

Learning Object Design and Sequencing

As the instructional designer involved in the development of the pharmacology learning object, I became interested in the implications of using a design methodology specifically prescribed for learning objects.

The Dick and Carey (1996) model, which uses a systems approach for designing instruction provided the actual framework for the development of the learning object. One of the best known models, its approach to designing instruction is similar to that of software engineering. The design model describes all the phases of an iterative process including; identifying instructional goals, analyzing learners and contexts, developing instructional strategies and assessments, and designing and conducting formative evaluation. However, the model does not provide specific support for decisions about the scope and sequence of learning objects like the Learning Object Design and Sequencing (LODAS) theory developed by Wiley (2000).

I was curious to find out if subsequently applying Wiley's Learning Object Design and Sequencing (LODAS) theory to the development of the pharmacology learning object would reveal different design decisions about the scope and sequence of the learning resources that were created.

The following steps as prescribed in LODAS were applied to the pharmacology learning object using the definitions and interpretations laid out in the theory; principled skill decomposition, work model synthesis, identifying the dimensionality of domain expertise, placing work models on scale of increasing complexity, synthesizing the integrated work models, and classifying the resulting work models and constituent skills (Wiley, 2000). This process was carried out in cooperation with the subject matter expert (Dr. Lawrence Spero, professor of Pharmacology at the University of Toronto) used to design the pharmacology learning object.

Qualification: Below is my first attempt at applying LODAS to a pretty complex subject (pharmacokinetics). As such, I can't profess that I have applied the theory as Wiley had intended or that I have represented the domain of 'therapeutics' correctly.

1. Determine appropriateness. The goals, values and conditions identified in the theory were deemed to be consistent with that of the instructional designer and the instructor. The goal for the learning object focuses on the development of a set of complex cognitive skills as described by Van Merriënboer (1997) cited in Wiley 2000. The desire is for therapeutic students to be able to predict the appropriate (non-toxic) use of a new drug.

-Complex, in the sense that they comprise a set of constituent skills that at some level involve conscious processing

-Cognitive, indicating that the majority of constituent skills are in the cognitive domain as opposed to affective or motor domain

The learning environment fosters a sense of cooperation between instructor and student where assessment serves to promote progress. The students are regarded as willing to monitor and regulate their own learning and have some competence in using the computer. Instructors are willing to empower students to direct their own learning and see the benefit in viewing the environment from the perspective of a student.

Analyze and Synthesize Content

2.Principled skill decomposition. The complex cognitive skill to be taught is broken down into its constituent parts.

•identify physical properties of drug (Pka partition coefficient, solubility, molecular weight)
•identify chemical (molecular composition) properties of drug
•determine where the drug works
•recognize how the drug works
•establish factors relating to the pharmacokinetic processes
- Absorption - route of administration, blood flow, surface area
- Distribution - relative blood flows, protein binding
- Metabolism - depends on chemical nature of drug & route of admin
- Elimination - route and process of excretion
•understand the relationship between the concepts (i.e. absorption, distribution, metabolism, and excretion of drugs) that underlie therapeutic principles.
•identify the age, gender, weight, height (sometimes), and disease state of patient
•history taking – involves taking xrays, blood work, etc…
•integrate information - pattern recognition from measurements
•evaluate info - compare experience to clinical practice guidelines

3.Synthesize work models. The constituent skills are recombined into activities that people perform in the real world.

i. Identify factors related to the structure of the drug
ii. Identify factors related to the process of the drug
iii. Recognize situation in which a therapeutic principle applies
iv. Identify interactions between physical & chemical properties with pharmacodynamic processes
v. Determine patient variables that influence biological response to drug.
vi. Measure the parameters of the disease
vii. Diagnose disease
viii. Prescribe drug

4. Identify the dimensionality of the domain. A model in the form of expertise representation is developed to illustrate what a learner can do with their knowledge. The domain of therapeutics involves the ability to predict the appropriate (non-toxic) use of a new drug and has the following domains of expertise.

a. Pharmacodynamics involves knowing where and how a drug works.
b. Pharmacokinetics involves knowing the therapeutic principles of drug administration.
c. Clinical Diagnosis. Involves determining the cause of a disease and prescribing a treatment based on the patient variables and the known effects of the drug.

5. Place work models on scale of increasing complexityA determination is made about which work models belong to each scale and what the relative difficulty of each work model is (see figure 6). The letters beneath each of the scales correspond to the work models identified in step 3.

6. Synthesize Integrated Work Models. The scales of expertise were divided into sections which approximate novice, intermediate, and expert levels of knowledge and were synthesized into work models that integrated skills across the dimensions of expertise (see coloured circles in figure 6).

(The resulting domain map still needs to be subjected to expert review to identify possible revisions.)

The work models, identify factors related to the structure of the drug (i), and determine patient variables that influence biological response to drug (v) were combined to form the integrated work model ‘drug structure and response’, represented by the blue circle (novice). The work models, identify factors related to the process of the drug (ii), identify interactions between physical & chemical properties with pharmacodynamic processes (iv), and measure the parameters of the disease (vi) were combined to form the integrated work model ‘pharmacokinetics’, represented by the green circle (intermediate). The expert-level performance on work models from all scales was combined to represent the complex cognitive skill originally identified in step 1. The work models of recognizing a situation in which a therapeutic principle applies (iii), diagnosing a disease (vii), and prescribing a drug (viii) were combined to form the integrated work model, "therapeutics", represented by the red circle (expert).

Design Practice and Information Presentation

7.Classify the work models and constituent skills. The components of the complete complex cognitive skill were classified into recurrent and nonrecurrent skills. Recurrent skills were those recognized as being executed in a similar manner each time. Nonrecurrent skills were those that differed widely according to the situation in which they were performed.

Work models associated with the pharmacodynamics domain of expertise are invariant. The chemical and physical properties of the drug do not change. The work models associated with pharmacokinetics are also classified as recurrent skills. The skills for recognizing the situation in which a therapeutic principles applies remains the same and is dependent on the patient variables and the interactions of the drug. However, the application of this information to the therapeutic domain of expertise is nonrecurrent as it is largely dependent on the clinical diagnosis.

As a result of this analysis the prerequisite knowledge for the recurrent skills was identified. Some knowledge of basic physiology and anatomy would be required along with a familiarity of the concepts of absorption, distribution, metabolism, and excretion of drugs.

Applying this model has given me a real appreciation for the quality of work that Wiley and other's have done in this area.

Future postings will discuss the implications of following this model.

Evaluation Results - Pharmacology Learning Object

I recently finished compiling the evaluation results of the pharmacology learning object.

Method

The participants consisted of 2 health science faculty and 10 students from the CLOE consortium. I used the Learning Object Review Instrument (LORI), version 1.4 developed by Belfer, et al. (2002) to collect faculty's individual assessments of the quality of the pharmacology learning object. Faculty were asked to assess the learning object in 8 categories using a five point rating scale ranging from low to high and to provide a rationale for their score.

A survey with 25 questions was distributed to students in order to carry out a learning impact study based on their use of the learning object. The student questionnaire items were combined into four sub-scales: Learning Value; Value Added by the Learning Object; Usability of Learning Object; Usability of Technology

For a more detailed description select evaluation methodology.

Results

The results from the faculty evaluation indicate that the learning object scored high in the categories of content quality, motivation and instructor/ student guides. It was also rated very good to high in learning goal alignment, presentation design and reusability. Interaction/ usability and feedback/ adaptation received lower scores ranging from good to very good and satisfactory to very good respectively.

Student Questionnaire results appear in Table 3. These results show that 75.7% and 77.5% of the possible responses within the Learning Value subscale and the Value Added subscales, respectively, were in the Agree and Strongly Agree categories. The responses for Usability of the Learning Objects and Usability of the Technology were in the Agree and Strongly Agree categories 80.0% and 92.0% of the time, respectively.

Student comments

The responses to the following questions that expressed the most common themes were recorded.

What did you like about this LO?

"I like the use of animations. I am a visual learner so any type of diagram is helpful, but animations are sort of novel and made the module more interesting."

"I liked the way it was broken down into the simplest components so you could figure it out yourself even if you still weren't sure about the material. Also, the fact that it is always there and you don't have to go out and get a CD."

"The interactive process and the use of the graphs related to drug effect on the body."

How could this LO be improved?

"Need to make output continue to relate to variables chosen by the learner. Should label diagrams a bit better for use beyond the demos. Should be able to have 'type of drug' selection as well, not just specific drugs."

"Sounds, more use of animation to support use of formulas. Narration. Glossary. Drug dictionary to describe what it is for.

Please use this space for any further comments you’d like to make about the LO including clarification of any of your responses.

"I would not prefer just a lecture on the topic. I feel that a lecture and a seminar using the LO would work well together."

"The equations page was kind of useless for a basic knowledge of pharmacology. It doesn't tell you what it's for and how it is tied in."

"I found that the LO improved understanding of pharmacokinetic principles."

Discussion of the Findings

The evaluations of the quality of the learning object completed by faculty indicate that it will be a valuable resource for instruction. It is interesting to note that the content quality and the instructor/ student guides received the highest rating and reflects the amount of work that was done to select the appropriate strategies for the learning object. The high ratings for motivation reinforce the constructivist notion that activities which engage the learner are perceived as being more relevant and therefore more applicable for students. The ‘very good’ rating for reusability was somewhat surprising given the amount of modifications that were identified in order for it to be used in other contexts. The ‘low score’ by the one evaluator in the area of feedback/ adaptation may indicate that more direction is necessary for generating the graphs.

The ‘learning impact study lite’ served to reinforce the assumptions about the target audience, and the prior knowledge required by students in order to use the learning object. The results of the ‘usability of technology’ subscale (Agree and Strongly Agree 92.0%) reflects the prevalent role that technology plays in student learning. The 75.7% and 77.5% (Agree and Strongly Agree) responses for the Learning Value subscale and the Value Added subscale suggests that the learning object will help students to better understand the therapeutic principles of drug administration. Most significantly, this reinforces the design decisions about the scope and sequence of the learning object. This also supports the rationale for developing a learning object that could convey the impact that a number of variables have on the overall effect of the drug and the achievement of optimum therapeutic effect.

References

Belfer, K., Nesbit, J.C., Archambault, A., & Vargo, J. (2002) Learning object review instrument (LORI). Version 1.4. http://elera.matchbox.surrey.sfu.ca/eLera/Home

The Reusability Myth of Learning Object Design

Jan 16. Having thought about it for a few days, I have added some additional comments which are identified with [edit] tags and italicized.

Stephen Downes - Design, Standards and Reusability

Below are some of my reflections on the concept of 'reusability' within the learning object paradigm. The above article is a useful starting point for becoming well-versed in the discussion. It may be-labouring the point, but I have yet to see this issue applied to a real example. I'm sure that I will refine my ideas for the paper I am working on, once I have a chance to dig deeper into the subject matter.

One of the first rules of instructional design is to identify a specific target audience. The better you are able to define this group in terms of their needs, pre-requisite skills and knowledge, the more likely you are to develop instruction that helps them to achieve the desired outcomes. It may be a moot point for those that are experienced in the area of instructional design but one that requires re-stating given all of the discussion around the re-usability of learning objects.

[edit]

To promote the adoption of learning object design based on the value implied by their re-usability is to propagate a myth about the practice of designing effective instruction. Designing learning objects that are at the level of being easily re-purposable are merely 'objects' and runs counter to best practices of instructional design because they are largely devoid of the principles that we know will enhance learning. Merrill (2000) hypothesized that, "when a given instructional program or practice violates or fails to implement one or more of these first principles, there will be a decrement in learning and performance.

[/edit]

The Issue

At the current time the only instructional design theory that I am aware of that addresses the issue of granularity and sequencing in learning objects is David Wiley's LODAS. He combines a number of existing instructional design theories; Elaboration Theory (Reigeluth, 1999), Work Model Synthesis (Gibbons, et al., 1995), Domain Theory (Bunderson, Newby, &Wiley, 2000), and the Four-Component Instructional Design model (van Merriënboer, 1997). The important distinction is that he extends these theories by addressing two fundamental issues in the design of learning objects: granularity and sequencing. This is a significant contribution, as it speaks to the very crux of the debate surrounding the use and 'reuse' of learning objects.

The promise of the learning object paradigm is based on the premise that economies of scale can be realized by designing digital learning resources at the level of a "common element" that can be shared with other institutions (Downes, 2000). Lower production costs result by revising or recombining existing resources instead of creating them from scratch. This practice has existed in various forms for a long time (it's one I used to regularly engage in as a secondary school teacher). What is novel about the digital frontier is its' promise of making this process easier by using technology to store information about the learning resource (meta tags) so that it can easily be retrieved and combined with like objects to produce a learning outcome.

While I believe that realizing cost savings is in itself a credible and pragmatic reason for designing learning objects with the above-stated affordances in mind, there comes a point where breaking an object down to its most 'common elements' is not desirable or practical.

I'm not debating the merit of what Downes is advocating but I am suggesting that it is a difficult proposition to implement. I fully believe in the value of sharing (that's why we've made our learning object available under a creative commons license), but not at the cost of compromising the instructional integrity of the LO we developed (elaborated on in the debate section).

The Context

My views on this issue are based on my experiences in developing a pharmacology learning object and as a former member of an educational consortium that designed and shared digital teaching resources. Increasingly we found that in order to make our electronic curriculum more relevant for classroom teachers (and therefore more reusable) we had to divide our material into modules so they could deliver the mini-lessons in their allotted computer lab time. Additional revisions were made by teachers prior to delivery to account for differences in technical setup, class time, learning rates and styles. That is to say our 'common element' still required a number of changes before it was pedagogically useful to instructors. Not to mention our concern that in order to achieve the 'common element' we had to strip way a lot of learning and therefore the inherent value of the resource.

Not only did this affect the way we designed subsequent resources, we ended up having multiple versions of the same material to satisfy different groups. I would argue that this consumed a lot of unnecessary resources and ultimately took us away from our intended audience - online distance education students. At that time (1997) webpages were not dynamically generated and therefore revising the content required considerable work (re-formatting pages, updating links, expertise in Toolbook Authorware, etc….). And then curriculum reform hit!!!!! Needless to say this project is no longer running - it was not a sustainable model. This approach may have provided value to those who used the resources (did not have to start from scratch), but it was a huge burden for those creating them.

Why is this context important? Because, I see the same pattern emerging with learning objects.

The Debate

I think our enthusiasm for the concept of 'reuse' in the learning object paradigm must be tempered with a more realistic appraisal of the environment in which we are currently operating.

Hamel & Ryan-Jones, (2002) and others advocate that, learning objects should be relatively small, citing the following passages.

While there is no optimal size for a learning object, it has been suggested that they be kept relatively small to increase the potential for reuse (Quinn & Hobbs, 2000), and to facilitate an adaptive, competency-based approach to training (Longmire, 2000). If each learning object is based upon a single enabling objective, and the granularity is small enough, then each learning object will be "appropriately" small.

Ideally it would be great if the pharmacology learning object we created turned out to be at a level that was common to other medical schools. However, I suspect this is not the case. Based on feedback of our LO from faculty and students their definition of a 'common element' for studying this subject is a moving target. Some, depending on their area of study, viewed additional information about drug categories as a necessity for studying this topic. Pharmacokinetics is taught in a cross-curricular manner in a number of different health science disciplines. The point being, that there are a number of different ways in which it is taught.

If as an instructional designer, I was to attempt to take into account the multiple contexts in which the topic was being taught in order to design a learning object as a 'common element' and in order to promote its reuse, the process would be halted in its tracks. In LODAS Wiley (2000) discusses using expert reviewers to "identify the dimensionality of domain expertise". This is an excellent idea and one which you would think would facilitate some common understanding. However, even subject matter experts have difficulty agreeing on this area.

My issue with the concept of reusability in learning objects is that it runs counter to instructional design best practices. I agree with Shaw (2002) who stated that in developing learning resources, one should begin with a genuine instructional problem and should strive to achieve outcomes which are not otherwise possible. The instructional challenge at each institution may not be the same and to be frank its not my concern. In order to do my job properly I need to attend to needs of the local students (the ones with which I am most familiar) in order to develop effective learning resources. Isn't that the ultimate goal? To reduce the pharmacology learning object to its most 'common elements' would be to re-create what already exists in the textbook using another media. Is this really what we want? Our reason for developing the LO in the first place was that we felt that the text book did not adequately represent or visualize the relationships which exist between therapeutic principles and the parameters relating to the physical and pharmacological properties of drugs.

Having said this we have attempted to make our learning object 'pedagogically reusable' to a certain extent…. Boskic (2003) made an important distinction in suggesting that there is a difference between pedagogical and technical reusability.

Pedagogical Reusability

I am inclined to agree with Downes that the actual content of the design is not as important as the approach to learning implicit in it.

That's why I included an external link to an instructor and student guide from the learning object. These guides were offered as a complement to the Pharmacology learning object and were intended to provide information that could enhance instruction. The strategies included were not prescriptive and instructors were not required to employ these methods when using the learning object. The guides included a description of the LO, learning objectives, target audience, pre-requisite knowledge and information on how the design of the LO supported the learning of pharmacokinetics. The goals in the guides are stated in observable terms in order to describe what will be accepted as evidence that learners have acquired the cognitive capabilities suggested by the goals. Because this information was not physically embedded within the pages an instructor could choose to use the LO to achieve learning outcomes different than the ones identified by me (i.e. to demonstrate issues of visual design to graphic art students). However, the fact remains that the strategies to reach the outcomes I identified are embedded within the LO as best practice of instructional design would dictate.

(you may want to look at the LO http://icarus.med.utoronto.ca/lo/pharmacology9/index.swf
at this point in order to understand the example below)

Eg. The first 3 learning objectives below correspond to an expository approach in which the “Basic Principles” tab was designed to demonstrate each principle graphically. These demonstrations illustrate how these rules can be used to explain, control and predict the effects of drug administration. The description that accompanies the animation is useful in explaining the ‘whys’ of the principle and makes it meaningful for the students.

1. The learner will be able to list and describe the major therapeutic principles of drug administration.

2. Given a demonstration of a therapeutic principle the learner will be able to identify and replicate the relationship between the concepts (i.e. absorption, distribution, metabolism, and excretion of drugs) that underlie the principle

In order to fully acquire a principle, the student must learn to apply that principle in a variety of new situations (Smith & Ragan, 1999). Using the ‘drug options’ tab the student can practice replicating the basic principles by selecting a range of patient variables, routes of administration and drug dosages.

3. The learner will be able to identify the relevant principles which describe the magnitude and direction of change plotted in the blood concentration time curve as well as a visual representation of the area under the curve (AUC).

The fourth objective corresponds to an inquiry approach in which the learner uses trial and error to learn about the basic therapeutic principles. Student can randomly select a range of patient variables, routes of administration and drug dosages and try to induce the principle which applies to that situation.

4. By manipulating the patient variables, routes of administration and drug dosage the learner will be able to correctly explain, predict and control the effect of these changes on the patient.

As a consequence this LO has limited applicability outside of the context for which it was designed. To strip out these strategies would be to take out most of what is valuable about the learning object leaving us with just an 'object'. It's what Wiley (2002) referred to as the 'reusability paradox' and has serious implications for the concept of reusability in the LO paradigm.

Technical Reusability

Given the technical sophistication of the LO we designed using Flash MX it would be difficult to change it to reuse it in another context or to achieve other learning outcomes. It has about 15 layers with different movie clips, action scripting to generate graphs based on user input and an equation, and a drug database (text file which potentially could be edited). I highly doubt that anyone would be interested in delving into the code to change either the principles we have demonstrated or to alter the images we have used. We also did not meta-tag the various constituent components that combined to make up this object, so that others could retrieve and re-combine the constituent parts into alternate configurations. This would have been an incredibly arduous process. I know Friesen
and company are working hard to streamline the meta-tagging process but as it stands this remains a major hurdle to overcome in the mainstream adoption of the LO paradigm. Others have documented how difficult a task this is including a presentation at last year's CADE conference by Dr. Elizabeth Murphy and Kevin O’Leary who shared their experiences in "Locating, tagging, and transferring learning objects" (online report not available).

As a result of the technical and pedagogical issues outlined above, most users would be stuck with reusing the object "as is". It may not be entirely appropriate for their instruction, but probably better than the alternative. There is the possibility that it could be combined with resources at Martindale’s Health Science Guide, a resource center listing 60,000 teaching files and 129,000 medical cases as described by Downes (2000). Students could use the pharmacology LO to make decisions about the administration of drugs referenced in some of their cases.

[edit]

I think what Downes is talking about is the difference between reusing and re-purposing. Re-purposing implies that some changes would have to be made to the actual content of the learning object not just the strategies used to achieve learning with it. As the above analysis demonstrates our learning object can be reused but not re-purposed without a great deal effort.

[/edit]

Alternatives?

Downes (2003) seems to be advocating an "environment where people decide for themselves what to do and when to do it. It is the difference between requiring a director and requiring a coach." This is consistent with a constructivist view of learning in which the student defines what they will learn and how they will go about learning it. As an educator, I see the value in learner-centred design but not all students prefer to learn this way or have the skills necessary to execute such a plan. For example, a radical constructivist would have suggested that it was inappropriate to propose goals for the pharmacology LO because educators do not know what learners' need or want to learn (Smith & Ragan, 1999). However, with a subject as complex as pharmacokinetics one can't assume that individuals who are novices in this area would be able to devise an approach to acquire the knowledge.

To facilitate the kind of environment Downes is proposing would mean having to design learning objects based on the "common element" approach so that individual objects could be combined rather effortlessly by individual learners in a bid to acquire new knowledge. As I have argued above this is an impractical and undesirable solution.

[edit]

Downes is correct in stating that creating learning objects with that level of re-purposability, "shoots an arrow straight into the heart of the discipline known as instructional design".

[/edit]

Some of the major factors which seem to be overlooked when discussing the design of learning objects is the motivational and social dynamics of engaging in this type of activity.

Previous LO instructional design analogies and metaphors (legos, atoms, music analysis) have focused on the technical aspects of the process. Below is one that builds into account social issues and motivational factors. One that needs to be addressed if the movement is to continue going forward.

Asking an instructional designer to create an LO based on the "common element" is like asking a local politician to develop municipal legislation with a provincial (state) and national level audience in mind. Ideally it would be nice for people to think in terms of the 'bigger picture', but it's not too realistic. The local politician would have no idea about the law's potential application outside the local jurisdiction (or the context in which it would be implemented), and would incur the cost of enacting the legislation, not to mention the amount of work involved in creating it (taking all things into consideration). After all is said and done, the final product would have to be watered down in order to accommodate different perspectives and political motives. The result is something radically different than what was envisioned at the outset.

Furthermore the local politician is accountable to the constituents of the district he/ she represents and therefore meeting their needs is the primary consideration. And this is where my metaphor breaks down… because in the political context a politician who meets the need of local constituents is likely rewarded with re-election. Such is not the case for faculty who for the most part do not receive appropriate recognition for the time invested in learning to use and develop instructional technology (for more on this issue see Culp, G. (2001) Faculty Rewards in Digital Instructional Environments. Syllabus Magazine).

Conclusion

In conclusion, I think we should acknowledge the fact that attempting to realize the full potential of "reusability" with learning objects has potentially undesirable consequences. We really need to ask ourselves whether it is worth the opportunity cost of making our learning resources so reusable as to strip them of their inherent value. Although, it is getting easier to retrieve material (using RSS) and share resources (CAREO, CLOE, MERLOT), teaching will continue to be a time intensive process requiring a great deal of thought and effort if it is to be effective. Designing for a global audience leads to different decisions about granularity and sequencing of learning objects and takes the instructional designer away from their initial goal - meeting the needs of their learners.

More time should be spent on exchanging best practices for designing and applying learning objects to instructional contexts than the content itself.

References

Boskic, N. (2003). Faculty assessment of the quality and reusability of learning objects. Unpublished Master's thesis, Athabasca University, Athabasca, Alberta.

Downes, S. (2000). Learning objects. Retrieved November 2002. Available:
http://www.atl.ualberta.ca/downes/naweb/Learning_Objects.doc

Hamel, C.J., & Ryan-Jones D. (2002) Designing Instruction with Learning Objects.
International Journal of Educational Technology, 3(1). Available: http://www.ao.uiuc.edu/ijet/v3n1/hamel/index.html

Longmire, W. (2000). A primer on learning objects. ASTD Learning Circuits, March 2000. Online: http://www.learningcircuits.org/mar2000/primer.html

Quinn, C. & Hobbs, S. (2000). Learning objects and instructional components. Educational Technology and Society, 3(2). Online: http://ifets.ieee.org/periodical/vol_2_2000/discuss_summary_0200.html

Shaw, S. (2002, February). Designing online facilities that really leverage learning. Paper presented at the Knowledge Media Design Institute, Interactive Webcast, University of Toronto, ON. [online]. Available: http://epresence.kmdi.toronto.edu/archived.asp

Smith, P.L., & Ragan, T.J. (1999). Instructional design. (2nd ed.). Toronto: John Wiley & Sons. Inc.

Wiley, D. A. (2000). Learning object design and sequencing theory. Unpublished Doctoral Dissertation, Brigham Young University, Provo, UT. Available: http://davidwiley.com/papers/dissertation/dissertation.pdf

Wiley, D.A. (2002). The reusability paradox. Available: http://rclt.usu.edu/whitepapers/paradox.html

Learning Object Evaluation

Evaluation

Three main strategies were used to assess the quality of the pharmacology learning object and to collect formative date for improving the resource. Early in the design stage the instructional designer conducted usability testing with a third year pharmacy student to obtain feedback on design and navigation issues. This informal meeting was conducted as a 'think-aloud session' where the instructional designer recorded the reflections of the student as she interacted with the learning object. Secondly, peer reviewers were asked to evaluate the quality of the learning object using an established rating instrument and to provide feedback for improvement using an instructor survey. Thirdly, questionnaires were distributed in order to carry out a learning impact study of students using the learning object.

Participants

The initial reviewer involved in the think-aloud session was selected because of her ability to provide feedback from the perspective of being a former pharmacology student and as a current pharmacy student (both were identified as target audiences for the learning object). Health science faculty and students from three separate institutions participated in the formal evaluation of the learning object. The institutions were members of the CLOE consortium and were committed to collaborating in such activities. None of the participants were required to have experience with learning objects or online resources. The only criteria for selection was that they indicated an interest in potentially using the learning object for instruction or studying.

Instruments

The Learning Object Review Instrument (LORI), version 1.4 developed by Belfer, et al. (2002) was used to collect faculty's individual assessments of the quality of the pharmacology learning object and to ensure that a consistent evaluation criteria was used by all participants. Faculty were asked to assess the learning object in the following areas using a five point rating scale ranging from low to high and to provide a rationale for their score.

1) Content Quality: Veracity, accuracy, balanced presentation of ideas, and appropriate level of detail
2) Learning Goal Alignment: Alignment among learning goals, activities, assessments, and learner characteristics
3) Feedback and Adaptation: Adaptive content or feedback driven by differential learner input or learner modeling
4) Motivation: Ability to motivate, and stimulate the interest or curiosity of, an identified population of learners
5) Presentation Design: Design of visual and auditory information for enhanced learning and efficient mental processing
6) Interaction Usability: Ease of navigation, predictability of the user interface, and the quality of UI help features
7) Reusability: Ability to port between different courses or learning contexts without modification
*8) Value of accompanying instructor guide: ability of resource to enhance instructional methodology


Source: Adapted from Belfer, et al. (2002)

*(The original items in the LORI which dealt with accessibility and standards compliance were replaced with the category in item 8 because the participants lacked the necessary knowledge to provide an assessment in both of those areas).

Slight modifications were made to the instructor and student surveys developed by Dawn Leeder (2003) of the Universities' Collaboration in eLearning (retrieved from http://www.ucel.ac.uk/resources/dev_pack.html 11/2003). These tools were used to provide faculty with an opportunity to suggest modifications for improvement and to provide evidence that the use of the learning object positively impacted student learning of pharmacokinetics.

Process

The evaluation of the pharmacology learning object and the learning impact study took place over a period of two weeks and was completed individually. The LORI was not used in combination with the convergent participation model for the evaluation of learning objects as proposed by Nesbit et al. (2002). The goal of using the instrument was not to increase inter-rater reliability and the process of a collaborative evaluation would have been difficult to manage and would have required more time of the evaluators.

The website location of the learning object along with the following electronic documents were sent via email to aid participants in the evaluation process.

1) A description of the learning object review instrument developed by Belfer, et al. (2002). This document laid out the criteria and scale to be used for evaluating the learning object. It also included examples of characteristics of low, average and high ratings for each category.

2) The learning object rating sheet where the results of the evaluation were recorded. Participants were encouraged to include the rationale for their rating in each of the specified criteria.

3) The learning object instructor survey which allowed the faculty evaluator to propose modifications.

4) An instructor guide was offered as a complement to the Pharmacology learning object and was intended to provide information that could enhance instruction. The strategies included were not prescriptive and instructors were not required to employ these methods when using the learning object.

5) The student evaluation survey containing 25 questions which were based on the students' use of the learning object and student guide. This formed the basis of the learning impact study lite.

6) The student guide. A resource for instructors wishing to provide training in specific strategies for using the learning object effectively.

The responses to the questions were entered into the electronic documents, re-saved and returned to me via email.

Future postings will reflect on the findings of this evaluation and the process itself.

References

Belfer, K., Nesbit, J.C., Archambault, A., & Vargo, J. (2002) Learning object review
instrument (LORI). Version 1.4

Nesbit, J., Belfer, K. & Vargo, J. (2002). A convergent participant model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28 (3). Available online http://www.cjlt.ca/content/vol28.3/nesbit_etal.html
For more information about the learning object review instrument that was used and the research behind its' development goto the Elearning Research and Assessment Network (Elera).

Vargo, J., Nesbit, J., Belfer, K., & Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and inter-rater reliability. International Journal of Computers and Applications, 25 (3).

Peer Review of Learning Objects

Part of the mandate of CLOE is to promote learning object design and evaluation as a scholarly activity. Therefore each object that is submitted to the repository will undergo a peer review similar in structure and process to the one used when submitting an article to an academic journal. The objects are evaluated according to clearly identified criteria by a panel consisting of an editor, subject matter experts and an instructional designer.

Subject matter experts will be responsible for examining the LO for the validity and quality of the content, for factual information and for the overall contribution of the LO towards student learning.

Instructional designers will evaluate the degree to which the LO is likely to meet its instructional goals as described by the author(s) and defined in the *Guidelines for Authors (below).

· Effectiveness as a Teaching/Learning Tool (Scale = not at all, somewhat, definitely)
· There are clear learning objectives.
· The learning object meets the stated learning objectives.
· The target learners are clearly identified (academic level addressed/technical ability/demographics).
· There are clear instructions for using the learning object.
· The technology helps learners to engage effectively with the concept/skill/idea.
· The learning object provides an opportunity for learners to obtain feedback within or outside the learning object.
· The author provides evidence that the learning object enhances student learning.[1]
· Pre-requisite knowledge/skills, if needed, are identified.
· The learning object stands alone and could be used in other learning environments.
· [1] Acceptable evidence could be anecdotal comments, student perception questionnaires, or more formal learning impact studies.

(source: CLOE Peer Review, November 2003 (PowerPoint Presentation from Dr. Michael Clarke, CLOE Peer Review Committee Chair, and Department of Microbiology and Immunology, Faculty of Medicine and Dentistry, University of Western Ontario) available online http://lt3.uwaterloo.ca/CLOE/CLOEPeerReview.ppt

This process is designed to ensure academic rigour and high quality objects. It may help institutions to evaluate faculty work in the area of instructional technology.

"…research on the impact of new computer technologies on the discipline is an important area of scholarship, and institutions need to encourage and reward faculty doing this kind of work."

Culp, G. (2001) Faculty Rewards in Digital Instructional Environments. Syllabus Magazine. Available online
http://www.syllabus.com/article.asp?id=410.

Co-operative Learning Object Exchange

Our learning object is being submitted to the Co-operative Learning Object Exchange (CLOE), "a collaborative project of seventeen Ontario universities developing an innovative infrastructure for joint development of multimedia-rich learning resources. The key innovation in CLOE is the creation of a virtual market economy for engaging multimedia to support online learning. Each institution will develop multimedia learning resources to address instructional challenges shared by the other partners. Each institution will contribute educational multimedia to the co-operative exchange and use resources developed by the other institutions in return."

As a member of this group we attended an Instructional Design & Learning Technology camp where we developed our skills and knowledge in creating interactive learning-centred objects. Each team was assigned an actual project and together we worked through the different stages of design; needs assessment, protoyping, sequencing/ scoping, usability testing, evaluation, project management and development. CLOE case story on designing re-usable learning objects.

CLOE is funded by eduSourceCanada, which promotes community building while developing learning objects

Hopefully, we will continue to expand the community of practice around learning object design and evaluation to include the sharing of code, instructional strategies, applications of LO's and evaluation methodology.

Instructional Guides - Learning Object

A link to an instructor and student guide are included with the learning object. These guides are offered as a complement to the Pharmacology learning object and are intended to provide information that can enhance instruction. The strategies included are not meant to be prescriptive and instructors are not required to employ these methods when using the learning object.

The decision to create a link to external guides was a conscious one. This makes the learning object more reusable because the strategies described in the guides are not embedded within the learning object itself. Instructors are able download the word attachments, revise them in order to meet their specific learning objectives.

The guides include a description of the LO, learning objectives, target audience, pre-requisite knowledge and information on how the design of the LO supports the learning of pharmacokinetics. Specific strategies for acquiring principles as outlined by Smith & Ragan, (1999) were also provided. In addition, some possible instructional approaches were suggested (ie. Case studies). A case study would require students to integrate learning from other courses such as pharmaceutics, physiology/ pathophysiology, pharmacology/medicinal chemistry and Pharmacotherapy I and II. Students would be engaged in thinking critically about the causes of the problems and could use the learning object to make decisions about the administration of drugs referenced in the case.

The purpose of including learning strategies in the student guide was two-fold. First, a set of effective strategies was provided to help them use the learning object to encode information so that it could be retrieved accurately. Weinstein (1978) found that students who received direct instruction in strategy use outperformed students who were merely informed that the strategies would be helpful. This involves teaching learners the procedures of the strategy and when and where to apply it. The strategies and process outlined in the student guide were designed to help instructors providing this type of training.

Second, by explaining the steps involved in processing the information, the student can be made aware of how this helps improve learning. Nkanginieme (1997) asserts that making a clinical diagnosis is the pivotal cognitive activity of a practicing physician. However, this process tends to remain a sub-conscious activity. By raising this type of activity to the level of consciousness one can be taught how to improve performance as well as acquire new knowledge.

References

Nkanginieme, K.EO. (1997). Clinical diagnosis as a dynamic cognitive process: Application of Bloom’s taxonomy for educational objectives in the cognitive domain. Med Educ Online, 2(1), 1-6.

Smith, P.L., & Ragan, T.J. (1999). Instructional design. (2nd ed.). Toronto: John Wiley & Sons. Inc.

Weinstein, C. E. (1978). Elaboration skills as a learning strategy. In H. F. O’Neil (Ed.), Learning Strategies (pp. 31-55). New York: Academic Press.

Learning Theory

The design of this learning object is based on Abdelhamid’s (1999) multi-dimensional learning model (MDLM) which depends heavily on illustrations and graphics. It integrates different memory strategies that require the student to think to generate information. His research into the process of learning and memory specify three main principles which enhance the understanding and recall of data.

1) The generation effect. A learner that generates an item is more likely to recall the information than when it is merely read (Houston, 1991). The pharmacology learning object requires the students to interact with the resource to create an outcome.

2) Spreading activation model. Stored information (represented by circles) is more easily retrieved when it is remembered in the context of data which is related and interconnected. Processing of one piece of information leads to the activation of the other related items.

The design of the learning object allows the students to see the relationship between the variables involved in drug administration and the effect this has on the patient.

3) Use of graphics can aid learning better than a verbal description. The animated demonstrations of the therapeutic principles enhances visualization of the affected systems.

source: (Abdelhamid, 1999)

In 1996, the School of Medicine at the University of Auckland conducted a study on the effectiveness of the multi-dimensional learning model. The results indicated that students’ academic performance was significantly increased in the areas covered by the model, while there was no significant improvement in the areas not covered by the model (Abdelhamid, 1997). In addition, the learners stated that they preferred this model for their medical education.

References

Abdelhamid, T. (1999). The multidimensional learning model: A novel cognitive psychology-based model for computer assisted instruction in order to improve learning in medical students. Med Educ Online, 1(1), 1-8.

Abdelhamid, T. (1997). An application of cognitive psychology in medical education using a specific educational program. Tarek's integrated system for learning and memory (TISLM): An evaluation of its effectiveness in improving learning and memory. Master of Literature-education University of Auckland.

Houston, J.P. (1991). Fundamentals of learning and memory. 4th ed. Florida: Harcourt Brace Jovanovich.