Ending an Online Quiz with “What’s Next” Instructions

Apr 4, 2013 at 10:34 pm, admin

Quizzes can take a variety of forms. I like quizzes best when they are formative, when the results of the quiz are used to improve or increase learning. The formative feedback from a quiz may be…

  • Corrective: explaining why an answer was wrong;
  • Recursive or remediating: sending learners back to previously studied (or skipped!) material or activities;
  • Elaborative: urging learners who scored high or answered correctly to explore the topic further, broader, deeper.

Online quizzing provides a fantastic mechanism for quick, automatic, and divergent feedback for learners. Quiz questions that can be assessed automatically by a computer should provide formative feedback on the question in general or even on specific answer choices, depending on what options students selected and whether the answer was correct or incorrect.

Canvas quiz results make it obvious what students got right or wrong, and provide differentiated feedback
Canvas quiz results make it obvious what students got right or wrong, and provide differentiated feedback.

Online quizzes can also let students re-take the quiz. Any quiz instance can consist of questions randomly selected and ordered from larger groups of questions, creating a potentially different quiz experience — and different question exposure — each time the quiz is attempted.

Unfortunately, by focusing on the summative nature of most assessments we’ve trained students to care about one thing in quizzing:

1. Scoring high

…well, maybe it’s two things:

2. Getting it over with

I wanted to direct students to review quiz question feedback so they would get more from the experience. Really this is about helping them understand that the quiz is about learning not just scoring.

I often want students to take the quiz again, too. Reminding them to do so at the end, just before they submit the quiz, might actually be encouraging, perhaps enough to even alter their attitude.

That was a long way of justifying why, in Canvas, I’ve started to include an Text (no question) “question” at the end of online quizzes that prompts students to review feedback or re-take the quiz if desired:

end prompt questions encourage students to review feedback and retake the quiz
End prompt “questions” encourage students to review feedback and retake the quiz.

I intentionally created these no-question questions in a Question Bank above the course level: at the Sub-Account or Account level, to be exact. This enables me — and other teachers/designers — to easily find and reuse these standard end prompts in any quiz in any course.

Finding questions from Account Question Bank in any Canvas course Quiz
Finding questions from Account Question Bank in any Canvas course Quiz

Does it work?

I don’t know yet, but it certainly could be tested. Imagine control and experimental groups, one without the end prompts and one with. The null hypothesis would be that the experimental group engages in just as many quiz re-takes and spends just as much time on quiz review pages.

It’d also be interesting to test students’ sense of self-efficacy after the quiz. Does reminding students that the quiz is part of learning change their sense of themselves?

O *is* for Outcomes. And Ongoing, Open, On-Target…

Feb 10, 2013 at 9:21 pm,

Scott Thornbury’s latest post O is for Outcomes gives a compelling and sympathetic argument against the current practice of testing (aka assessment) and outcomes-based assessment. I began drafting a comment on Scott’s blog, but it quickly expanded into a post of it’s own:

We can hate the idea of assessments and outcomes, or we can hate how they are predominantly designed, standardized, and implemented. We can hate both, but I find it too easy to conflate the two from our exasperation with the latter.

I sympathize with this point of view; I have seen far too many examples of bad assessment:

  • Assessments that do not connect to the real world situations that learners will find themselves
  • Assessments that do not provide useful feedback
  • Assessments that are not followed by opportunities to apply that feedback to improve learning

I have also seen more than enough examples of bad outcomes:

  • Outcomes for outcomes’ sake
  • Outcomes that are forced into taxonomy-driven verb statements because That’s The Way To Do Things And We Must Look Sophisticated Not Ignorant
  • Outcomes that do not encourage diverse, meaningful expressions of learning

Some colleagues have suggested to me that there are plenty of times when you don’t want stated outcomes at all. I have heard, for example, that instead of knowledge, understanding, etc., you may want learners to explore their own interests, embark on their own learning paths, engage with real-world networks and communities, or express themselves to their fullest.

Those sound like outcomes to me. Measurable? In some sense. Scott wrote: “An obsession with pegging learning to preselected and minutely-detailed outcomes now pervades every aspect of education”, because such outcomes seem to be more easily measured. But all that glitters is not gold, and many things that are easily measured are not worth measuring.

It may not be the principles of assessment and outcomes that fail, but rather a failure of instructional design or implementation.

To keep myself both sane and honest to learners, I continue to turn back to educators and researchers that I trust. For example Bloom himself (and not just interpreters) and his designs for mastery learning, and Grant Wiggins, especially Educative Assessment, etc.

I began teaching in the same field that Scott works: ESL/EFL. For much of the past ten years, however, I have taught web design and development as an adjunct. Both language teaching and web development aim to develop fluency and skill in learners so they can apply learning in novel situations, and ultimately achieve more in their lives. Knowing that this discipline very clearly aims to prepare learners for vocational success, I take outcomes and assessment quite seriously. At it’s heart, the thoughtful employment of outcomes and assessments helps me as a teacher — and students as learners — address the question, where are we going and how are we doing? If education is supposed to help people become more successful at life, can’t we–shouldn’t we–attempt to measure education’s ability to invest students with skills and knowledge that do open doors, empower, and build independence?

Removing Barriers to Openness with Canvas Network

Nov 1, 2012 at 10:36 am,

Forgive me if the following sounds like evangelism; this is a pretty exciting morning.

Late last night, Instructure launched Canvas Network, a new index of online, open courses built and delivered on the Canvas learning platform. There are several news items on this launch (e.g. TechCrunch, CampusTech, Hack Education, ZDNet), but here’s my own perspective:

Since I got involved in the open ed movement during my time at Utah Valley University, I’ve believed that the best way to practice open ed is not through heavy investment, but from heady inspiration, not through grant funding, but from the grass roots. The biggest barrier to engaging in open ed for many teachers is simply the act of re-publishin their online materials on a separate platform. At UVU we tried to reduce that barrier by offering a MediaWiki instance, a UMW-inspired WordPress instance, and a clunky (but inspired!) Moodle mod called OpenShare.

Canvas has always supported open teaching and learning practices at a course level. This is important, because Canvas has enabled teachers to make the choice about sharing openly, regardless of any institutional or grant-funded initiative. And not just the teachers who are willing to use open-oriented platforms, like wikis and blogs, but the more mainstream LMS users as well.

Canvas has also aimed to empower both teacher and learner by making powerful, modern web tools easy and quick. Powerful, modern, easy, quick — it’s deceptively simple — a MacGuffin, even. I often tell the story of how I discovered these traits were key to any bringing any significant and far-reaching change to the institutional status quo of teaching and learning. If a significant proportion–let alone the majority–of a school’s faculty isn’t leveraging technology to improve teaching and learning, there’s something wrong somewhere. Could be leadership. Could be culture. Could be technology.

I’ve found that Canvas the platform reduces the friction and lowers the barriers to technology-enhanced teaching and learning. Now Canvas Network aims to do the same for open education practices, by helping teachers and schools open up course materials and experiences to the world, on their own terms.

Canvas is a true multi-tenant cloud service, one that lets individual teachers teach (and students learn) for free. Thus, Canvas Network can instantly put the power of openness into the hands of tens of thousands of teachers, to share their instruction with millions of learners who are already Canvas users.

That’s pretty cool. It’s enough to make me reconsider earlier thoughts that open education is a niche philosophy with an even more limited practice. Only time will tell. But just as exciting as the prospect that the future of education could be fundamentally open, is my wonder at what schools and teachers and learners themselves will do on Canvas Network. Because we’re not setting the rules, that part of the future, too, is open.

Gardner Gampbell’s OpenEd12 Keynote

Oct 29, 2012 at 6:00 pm,

Gardner Campbell‘s OpenEd2012 keynote was overwhelming in it’s scope, depth, and general awesomeness. It deserves at least two viewings…

… and equal time in reflection and discussion.

Review of RSVP Reader Apps for iOS

Oct 15, 2012 at 6:07 pm,

I’ve long been interested in the progression of rapid serial visual presentation (RSVP) to facilitate certain types of reading. RSVP was originally pursued as a technological solution to speed limits on reading imposed by eye movement. See, in addition to the time required for eye-to-brain processing of words during fixations, there’s also the time taken by saccade movements when your eyes jump from one spot in a text to another. You can only move your eyes so fast. This explains, in part, why you’ll eventually hit a ceiling with reading speed.

RSVP attempts to solve this by displaying words in quick succession, one or two at a time, flashed in a single space on the screen.

Because RSVP never requires more than 5-20 characters on screen at a time, it didn’t take long for researchers to recognize that RSVP is a perfect fit for mobile devices.

I’ve been looking for the perfect mobile reader for some time, and while I still haven’t found the One Reader, I have discovered some good tools. Here’s a quick review of the better RSVP reading apps for the iOS:

Gotham Reader

App: https://itunes.apple.com/us/app/gotham-reader/id420684476

Gotham Reader integrates with Google Reader to fetch the full text of articles you’re subscribed to through RSS or Atom. Gotham Reader can’t open files or clipboard text.

And while loading Gotham Reader is painfully slow to start (it can take several minutes to sync each time you open the app), the settings and performance in reading are good.

I really appreciated Gotham Reader’s synopsis screen, which shows the first line or two of the post, along with all pertinent metadata, and an estimated time to read at the current rate.

Curiously, Gotham Reader lets you choose 2, 3, or 4 words per screen–but not 1. Type size can be controlled, but color can not.

Interface controls were intuitive: tap the article to start playing, tap to stop, which provides additional options, including speed.

Gotham Reader lets you save articles to finish later, or to store in Instapaper. You can also share any article.

What’s missing are labels for the rate and range of the display speed (this is simply indicated by a bar).

All-in-all, Gotham Reader is a solid RSVP reader, and indeed the only real RSVP reader that deals with RSS feeds with any kind of completeness or efficiency.

Simian Speed Reader (Speed-R)

App: https://itunes.apple.com/us/app/simian-speed-reader/id296829522

Simian Speed Reader provides RSVP display of TXT files. That’s all. But that’s really not terrible, considering Project Gutenberg. Still, I’d hope for ePub or PDF at the very least.

Simian lets you choose between 1-4 words per screen, and lets you select the display speed between 30 and 600 words per minute. I think the display speed is based on 1 word per screen, not actually on a calculation of the number of words per minute.

Simian lets you choose font as well as font color and background.

The RSVP reading experience in Simian is fairly stable, but note that the text size changes to maximize the size of the words on the screen. This can be disconcerting, so be sure to change the type size in settings down to a comfortable and consistent size if this bothers you.

There is also currently a bug that fails to account for punctuation when counting words. So, three-word phrase like “James said, ‘I” will show if you have two words set as the default.

Interface controls were simple, but limited: a Pause/Play button at the bottom must be hit to stop (whereas you’d hope to simply tap the screen anywhere).

I really want to like Simian Speed Reader, but it comes with a fundamental flaw: There is no Cancel, Done or other method of exiting Settings. This means you have to get out of the app and reopen it to return to the files menu.

Thankfully, Simian remembers exactly where you left off for each file.

iRSVP

App: https://itunes.apple.com/us/app/irsvp/id523989697

iRSVP provides RSVP display for files in a variety of file formats: “TXT, EPUB, RTF, DOC, DOCX, and limited support for PAGES ’09 and PDFs”. But this is even cooler: iRSVP’s Manage Files menu lets you add directly from Project Gutenberg by browsing and adding files within the app. iRSVP also supports clipboard pasting.

iRSVP Preferences lets you choose text color and background, and adds the useful ability to control punctuation delay speeds.

In addition to Pause/Play controls at the bottom of the display, iRSVP adds Previous Word and Next Word to facilitate review.

When paused, iRSVP shows your progress through the document, and lets you adjust display speed.

Like Gotham Reader, iRSVP fails to label the rate and range of the display speed (this is simply indicated by a bar).

iRSVP does not let you choose number of words or characters to display. You’re stuck with 1 at a time. This is iRSVP’s fatal flaw, I think, and I’ll elaborate on this in summary below.

LazyEye

App: https://itunes.apple.com/us/app/lazyeye/id415180387

LazyEye is an RSVP display reader for text pasted in from the iOS clipboard. Though LazyEye provides links to several content providers (e.g. LongReads), you still have to deal with select, copy, paste.

LazyEye’s Settings lets you slow down or speed up for punctuation, short words, or long words, though these are simple binary settings.

Each time you enter LazyEye the app reloads the current clipboard, making it impossible to pick up where you left off.

Once inside a text, LazyEye’s interface controls were intuitive: tap the article to start playing, tap to stop, which provides additional options, including advance or retreat one sentence or one paragraph.

Speed is controlled as words per minute, from 30 – 200.

LazyEye does not let you choose number of words or characters to display. You’re stuck with 1 at a time.

The paid version, Power Reader, seems to only remove ads and up to 1000 WPM. But with the clipboard limitation, I don’t know why you would bother.

Discussion

There are some obvious gaps in each of the RSVP reader apps reviewed here:

None of the apps accepted files directly from other apps, though they easily could. Most did accept text from the clipboard.

None of the RSVP reader apps were smart enough to use an if/else calculation to determine number of words based on visible characters. RSVP reading is based on the understanding that your eyes have a limited acuity, and the maximum range of focus in any fixation point is probably counted in characters, not words. True focus is about 4-5 characters, and some have estimated that since fixations can be 20 characters in length, the optimal number of characters is somewhere in between. An RSVP reading app would presumably be able to process a word, and if the number of characters is less than some amount (say, 12), it checks the next word. And so on. If the combined character count is less than the threshold, the reader displays both words. Otherwise it displays the one word, and moves on, repeating this check.

Nor did any of the RSVP reader apps show passage context on pause. It makes sense to show the words around the current highlighted word, perhaps in gray rather than white/black to show space. This, along with previous/next word/sentence tools such as those in iRSVP and LazyEye, might help address RSVP-related reading issues, such as attentional blink and task switching costs.

Finally, none of the RSVP reader apps built in any kind of note taking tools. At a minimum it would be nice to add notes at any point along the “timeline” of reading , i.e. when paused.

Obviously I have my own list of must-haves and nice-to-haves in an RSVP reader. But I’m fairly pessimistic about future developments, wondering if RSVP isn’t doomed to be the Dvorak keyboard of reading: a great idea with passionate advocates, but one that just can’t swim into the mainstream. Let’s hope I’m proved wrong by future iterations to these apps, or newcomers to the field.

Summary

While RSVP seems a perfect fit for reading on a small screen, there’s no one great RSVP app for the iOS. Gotham Reader is the best app of the handful reviewed here, but it works only with RSS fed content. There’s no great option for reading files. Though Simian Speed Reader and iRSVP are the best options, both suffer from some fundamental flaws. LazyEye is not yet worth your time.

Weekly Notes for 2012-10-07

Oct 7, 2012 at 8:49 pm,
  • Hey all, @jimgroom's post on open architecture, open courses just went live at http://t.co/kUPOT01Z #keeplearning #
  • Why I got Fired from Facebook (a $100 Million dollar lesson) http://t.co/VHltt0zn via @noahkagan #
  • @marionjensen We all know you have a secret relationship with the prez. Now make good on some favors! #
  • Halloween joke of the day: Do zombies eat popcorn with their fingers? No, they eat the fingers separately. #
  • @holden @ryanbrazell I always thought "undecided" was just a way to get attention. #
  • @holden @ryanbrazell But that's as close as I get to talking about tonight's debate. #
  • @holden That's alright. I don't mind political division or even fiery conversations. Just averse to them over the interwebs. Less empathy. #
  • RT @clsaarinen: Looking for @Instructure's new blog? Keep Learning, edited by @jstein. http://t.co/ztj2NChF < Needs your voice! #
  • @tjbliss Nice! Theory to practice, Bam! #
  • Body says, "You took one too many spills skating last night." But spirit says, "Take more, while you still can!" #
  • Did I really just read a peer-reviewed journal article that overtly conflated correlation with causation? #yep #
  • @lottruminates it's rarely as blatant. #
  • @lottruminates Sounds like you could start a good web site documenting the really bad research. #
  • @lottruminates As for the Slate article, isn't that covered in the first week of stats courses? ;) #
  • @lottruminates Sure thing. Good points were made. Like the arbitrariness of statistical significance. But of course, that can be tweaked. #

Powered by Twitter Tools

Richard Talbert on Learner Independence

Oct 3, 2012 at 5:49 pm,

Learner independence is tied to so many valuable aspects of learning, from critical thinking to lifelong learning (I’m especially interested in how it supports habits of deliberate practice and dwelling in flow states). This is a huge topic that I won’t even attempt to address here. But I do want to reference Richard Talbert’s latest post, “Declaring Independence”, which comments on the experience of teaching a flipped class, and concludes:

…helping people become become independent learners, capable of managing and directing their intellectual growth over their entire lifespans, is the fundamental goal on which higher education — maybe all of education — needs to focus.

New Community-Contributed Blog “Keep Learning” Goes Live

Oct 2, 2012 at 5:45 pm,

Keep Learning is live, with the inaugural guest post on open architecture and open courses by Jim Groom published at 1:06pm EDT.

Keep Learning Blog

Keep Learning is a new education technology blogging project by Instructure, the folks behind Canvas. It’s a privilege to have a chance to work on this project, and hopefully engage a large swathe of the varied education and technology communities. The project relies on guest writers and cross-disciplinary readers to build bridges to different shared spaces, professional networks, and online learning communities.

Add Disqus to a Page with Javascript

Sep 27, 2012 at 5:57 pm,

I’ve had some folks ask me how to add a Disqus discussion area to an existing web page via an external Javascript–especially on a page that may be within an LMS or content management system–after the page has loaded.

This is easy. Simply use the Disqus “universal code” Javascript, which I provide with some additional documentation below.

Of course, any usage of Disqus requires that you first register your own account and create a forum. This will provide you with the forum shortname that’s required in the documentation below.

After that, here’s the code you would add to your external JS file:

//Disqus
/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
var disqus_shortname = 'forumshortname'; // required: replace example with your forum shortname
//must turn on developer mode if you want secure sites to load this
var disqus_developer = 1; // developer mode is on
/* * * USAGE * * */
/* Within any page or tool, Switch Views to show the HTML and insert the Disqus  */
/* provided DIV element with a non-breaking space (&nbsp;) inside, e.g.: */
/* <div id="disqus_thread">&nbsp;</div> */
/* * * DON'T EDIT BELOW THIS LINE * * */
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'https://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();

Notes:

  • This snippet sets Disqus in developer mode (the universal code doesn’t do this)
  • Users visiting the page may be prompted to bypass browser security in order to see the content. On Chrome this is indicated as a very subtle shield in the address bar.

Directions:

  1. Add the above snippet to an external Javascript that is loaded by your system
  2. But first you have to change “forumshortname” to the shortname of your forum!
  3. In any content page or tool, switch to HTML view and add the following element: <div id=”disqus_thread”>&nbsp;</div>

Devlin Daley’s Ed Startup 101 Q&A

Sep 12, 2012 at 6:45 pm,

Of course I’m biased, but I thought Devlin Daley did a great job explaining the vision and foundation of Instructure and Canvas during today’s Ed Startup 101 Q&A. Hearing Devlin talk about Big Picture type things is always reinvigorating for someone like me who has the privilege of working with him and the rest of the Instructure team. Devlin has the rare ability to dive deep below the surface of technology challenges, refuting a superficial answer with a far more elegant, holistic solution.

Here’s the video:

Here are few salient points and quotes from the Twitter stream: