Introduction to Vocareum Virtual Labs

March 26, 2020

Webinar

Sanjay SrivastavaCEO

David LinBusiness Development

David Joyner, PhDGeorgia Tech

Agenda

• Vocareum Introduction - Sanjay Srivastava - 10 mins

• Teaching Computer Science at Scale - David Joyner - 30 mins

• Q&A

Mission

Deliver virtual labs and assessments

securely, on budget, and at scale

( incl. skills-based exams)

(infrastructure, tools, content)

(user roles, policies, spend)

(~ 500K learners)

Academic Partners

… over 600 institutions using Vocareum Labs today …

University of Illinois

RMIT University

Indiana University

Delft University

Supporting broad range of subject areasData Science

Fully hosted notebook solutions;Pre-installed packages

Programming

Broad range of languages; Standard IDE, spyder, eclipse.

Engineering

Stream desktop & applications; License flexibility, Raspberry Pi

Big Data

Hadoop processing;Run MapR on Clusters

Machine Learning

Any ML framework; Dedicated GPU

Cloud Computing

Policy controlled, budgeted; Aggressive resource mgmt

Database

MySQL, PostgreSQL;Neo4j, DynamoDB

Full Stack

Front- and back-end services;Business logic; Databases

DevOps & Cyber Security

Range of tools and infrastructure;Linux; Sudo/Root/Admin

C/C++

Assessment Options

ManualGrading

AutomatedGrading

In-ContextFeedback

PeerReview

MasteryLearning

CompetitionLeaderboard

Using diverse architecture to support labs• Web Applications

• Streaming Linux Applications

• Batch Compute Clusters

• Cloud Infrastructure

• BYO Applications + Infrastructure • Network of Docker Containers• Network of Virtual Machines• AppStream

Open source web applications/frameworks• Jupyter Notebook

• R Studio

• Terminal

• MySQL

• MongoDB

• Neo4J

• Postgres

Supporting a variety of languages:• C, C++, Python, R, Java, Scala, Go, PHP, etc.

• Hadoop applications, including Spark

• ML frameworks - TensorFlow, PyTorch

• Eclipse Theia

• Cloud 9

• Node

Open source desktop applications • MySQL Workbench

• Eclipse

• Rattle (for R)

• Spyder

• Firefox browser

• pgAdmin

Others can be easily added upon request

Learning Management & Assessment

• Enrollment

• Hosting Course Material

• Grading

• LTI integrations

Supporting Exams• Synchronous exams with 500 students• Asynchronous exams (90 minutes over 3 days)• Scheduled proctored certification tests• Exams with a passcode provided by proctor• Check whether learners came from a specific IP

range• Deliver random tests from a bank of problems• Built in plagiarism detection tests.

“I can see how it is very challenging to scale up the system so all students can access and work on the exam at the same time.

In previous years, we used our own online platform to handle students’ submissions. But, we found Vocareum to be much more maintainable and has better features. Using Vocareum really helps us: we know who submitted, at what time they submitted, and then we can run an auto-grading job immediately.”

Dr. Oka Kurniawan, SUTD

CONFIDENTIAL

Student Lab Interfaces

Supporting Advanced Coursework & Research• Long running jobs• Large data sets• Additional resources

• Cloud - AWS• Databases - MySQL, Mongo, Neo4j

• Aggregated labs + tools

Deploying Vocareum

Hands-on Labs / MOOC

Labs for learners to use tools, technology and services in managed environment (policy, budget, etc)

Skills-based Exams

Administer proctored skills-based certification exams, with automatic scoring

Training / Bootcamps / Hack-a-Thons

Course material + lab environments to support instructional training events

Research

Team-based lab environment with data security, grant management, etc.

Syndication

Course material + labs packaged and distributed to Universities, partners, etc.

Virtual Campus Labs

Upgrade on-prem computer labs with purpose-built cloud labs and tools

Security

Public Subnet Private Subnet

servers

distributed file system

object storage

proxy server

internet gateway

NAT gateway

Customer VPC

REST API

CONFIDENTIAL

Thank you!

Vocareum Webinar SeriesDavid JoynerMay 28, 2020

CS1 Online & At Scale

Goal:Scale up the availability of

intro-level computer science education.

Solution:Georgia Tech's online CS1301: Introduction to Computing in Python

course, available as both a massive open online course (MOOC) and a for-credit offering.

CS1301Introduction to

Computing

CS1301Introduction to

Computing Online

Course Videos

Integrated Exercises

Programming Exercises

Adaptive Textbook

Problem Sets

Proctored Exams

Recitations & TAs

Course Videos

Integrated Exercises

Programming Exercises

Adaptive Textbook

Proctored Exams

Recitations & TAs

~500 2-minute videos

~500 integrated multiple choice and text entry exercises

~200 integrated programming exercises with live feedback

Textbook with ~1000 multiple choice and text entry exercises

~150 programming and ~500 multiple choice/ text entry problems

Proctored exams occurring within a browser-based IDE

Traditional in-person recitations, graders, and teaching assistants

Problem Sets

Course Videos

Integrated Exercises

Programming Exercises

Adaptive Textbook

Proctored Exams

Recitations & TAs

Free MOOC

Verified MOOC

Certificate MOOC Series

Georgia Tech Course

Problem Sets

Free MOOC

Verified MOOC

Certificate MOOC Series

Georgia Tech Course

Opened February 2017, Free

Opened February 2017, $100

Opened July 2018, $400

Opened January 2017, Tuition

Honors Program Course

Dual Enrollment Course

Opened August 2017, Tuition

Free MOOC Series Opened July 2018, Free

Opened January 2018, Free*

Video CoursePlatform

Assignment PlatformProctoring Platform

Adaptive TextbookPlatform

Course Content

Partnerships

20% Headshot 80% Screencapture

All Course Content Available at bit.ly/CS1301PC

Video Content

Programming Problems Multiple Choice & Text Entry

All Course Content Available at bit.ly/CS1301PC

Exercises

Integrated Exercises

All Course Content Available at bit.ly/CS1301PC

Random Test Cases

All Course Content Available at bit.ly/CS1301PC

Sample Answers

All Course Content Available at bit.ly/CS1301PC

Timed, Proctored Tests

All Course Content Available at bit.ly/CS1301PC

Total Volume:Over 500 coding problems

All Course Content Available at bit.ly/CS1301PC

Automated Evaluation

All Course Content Available at bit.ly/CS1301PC

Phineas (Global

Autograder)

Student Code

Ferb (Local Autograder)

Grade

Feedback

Results

2017 2019

J F M A M J J A S O N D J F M A M J J A S O N D J F M

2018

Georgia Tech Course

Free MOOCFree MOOC (Series)

Honors Program CourseDE Course

Verified MOOCCertificate MOOC (Series)

A M J J A S O N D

0

200

400

600

800

1000

1200

1400

1600

1800

Enrollment Over Time (Georgia Tech)

2017 2018 2019

0

50000

100000

150000

200000

250000

Enrollment Over Time (All Versions)Georgia Tech Free MOOC Verified MOOC Free MOOC Series Verified MOOC Series

2017 2018 2019

Georgia Tech, 2069

Free MOOC, 678

Verified MOOC, 239

Free MOOC Series, 3300

Certificate MOOC Series, 5564

Total Course Completers by Version

Research Findings

Students in CS1301 online class perform as well as1 or better than2 students in a

traditional version of CS1301 in a pre-test/post-test setup.

1: Joyner, D. A. (2018). Toward CS1 at Scale: Building and Testing a MOOC-for-Credit Candidate. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. London, United Kingdom. ACM

Press.

2: Joyner, D. A. & McDaniel, M. (2019). Replicating and Unraveling Performance and Behavioral Differences between an Online and a Traditional CS Course. In Proceedings of the ACM Global

Computing Education Conference (CompEd). Chengdu, China. ACM Press.

Research Findings

Students who previously failed or withdrew from a CS class prefer the online version,

and perform better in the online version than the traditional version1.

1: Joyner, D. A. (2018). Intelligent Evaluation and Feedback in Support of a Credit-Bearing MOOC. In Proceedings of the 19th International Conference on Artificial Intelligence in Education. London, United

Kingdom. Springer.

Research Findings

Students rate the online version of CS1301 as higher quality than other college courses

they have taken1.

1: Joyner, D. A. (2018). Toward CS1 at Scale: Building and Testing a MOOC-for-Credit Candidate. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. London, United Kingdom. ACM

Press.

Awards

Bronze Award: Computing & IT Discipline2019 Reimagine Education Awards

Class Central Top 100 Course ofAll-Time (2019)

For more:bit.ly/CS1301PC

(the whole course in MOOC format)

LucyLabs.gatech.edu(research on the course)

DavidJoyner.net(me!)

Questions?