mlipcli

A CLI tool for managing workflows in the Machine Learning in Production course across Canvas LMS, Google Sheets, and Slack. Handles student rosters, grading assignments, submission downloads, grade management, and automated monitoring.

Installation

Run the project with npx mlipcli <module> <command> [options] (Node.js 24+)

If you prefer to run from source, use deno: deno run --A src/main.ts

This project needs a number of configuration secrets, that can be set as environment variables or in a .env file in the working directory:

Canvas (for Canvas commands):

• CANVAS_API_URL - Canvas API URL (e.g., https://canvas.cmu.edu/api/v1)
• CANVAS_API_TOKEN - Canvas API token, created in the Canvas profile: https://canvas.cmu.edu/profile/settings
• CANVAS_COURSE_ID - Canvas course ID, the number in the URL of the course

Google Sheets (for spreadsheet commands):

• GOOGLE_CREDENTIALS_PATH - Path to Google service account credentials JSON
• SPREADSHEET_ID - id of the Google spreadsheet that stores various data (service account must have write access)
• GOOGLE_DRIVE_FOLDER_ID and SPREADSHEET_NAME - Alternative to SPREADSHEET_ID, identifying the spreadsheet by folder and name.

Slack (for Slack commands):

• SLACK_ACCESS_TOKEN - Slack bot token

LLM (for breakout feedback commands):

• LLM_API_KEY, LLM_MODEL, LLM_API_ADDRESS

Important Modules

grade - Grading Operations

Download Submissions.

# Download submissions assigned to a specific TA
mlipcli grade download jdoe I1

# Download all submissions for an assignment
mlipcli grade download all I1

Clones and checks out the repositories at the commit the students submitted. If the submission has been graded, it creates/overwrites the grade.md file.

Creates a directory structure:

grading/
  I1/
    andrewId/           # Git repository cloned at submitted commit
      grade.md          # Rubric template for grading
    andrewId_resubmission/  # If student resubmitted
      grade.md

Rubric format. Each rubric item has a description followed by checkbox options with points. Mark exactly one option with [x]. Other markings like [?] can be used for temporary markings that still need to be checked. Lines starting with // are treated as internal comments not released to students. When giving 0 points, an explanation after the colon is required.

We can install and run your implementation based on the README.

- [x] 20p: Requirements met
- [ ] 0p: Requirements not met

No private credentials are committed to the repository.

- [ ] 10p: Requirements met
- [x] 0p: Requirements not met: API key found in config.js

// internal comment, visible to TAs, but not released to students

The grade check command checks that the rubrics are filled out correctly for all downloaded solutions. The check command validates each rubric item:

• Exactly one option is checked (not zero, not multiple)
• If 0 points is checked, a comment explaining the deduction is provided
• No comments on unchecked options (likely a mistake)

# Validate all grade.md files are properly filled in
mlipcli grade check I1

The grade report provides a breakdown of grades across all submissions.

Upload Grades.

Upload all grades from local grade.md files to the spreadsheet. This does not automatically release the grades.

# Upload grades from local grade.md files to spreadsheet
mlipcli grade upload I1

# Upload with automatic commit hash insertion
mlipcli grade upload I1 --addHash

# Upload specific students only
mlipcli grade upload I1 student1 student2

Release Grades. Instructors only: Instructors can check and release grades, which posts them to Canvas (after possible point penalties). Note that the grades are posted with a comment that is associated with the account pushing the grades.

# Release grades from spreadsheet to Canvas with comments
mlipcli grade release I1

# Release specific students only
mlipcli grade release I1 student1 student2

Releasing grades:

• Posts the grade to Canvas
• Posts a comment with the rubric
• Sends a Slack notification to the student

roster - Course Roster Operations

Manage student roster synchronization between Canvas and Google Sheets.

# Update roster from Canvas (marks removed students, adds new ones)
mlipcli roster update

# List all non-dropped students and TAs
mlipcli roster list

gradingassignment - Grading Assignment Operations

Assign TAs to grade students for specific assignments.

# Create grading assignment column and assign TAs to students
mlipcli gradingassignment init I1

The algorithm assigns TAs to students while:

• Balancing workload across TAs
• Avoiding conflicts (defined in spreadsheet)
• Trying not to assign the same TA to a student repeatedly

monitor - Submission Monitoring

Instructor only: Long-running process that monitors Canvas for new submissions and comments and updates the spreadsheet with the latest submission URLs (from where the grade command can download them). This also handles late days, token use, and regrade requests.

# Start continuous monitoring (checks every 10 minutes)
mlipcli monitor start

# Run a single check without processing (dry run)
mlipcli monitor test

# Process all pending items once and exit
mlipcli monitor once

The monitor handles:

• New submissions: Records submission URL, handles late tokens
• Resubmissions: Validates tokens, notifies graders
• Invalid URLs: Notifies students via Slack and Canvas comment
• Regrade requests: Notifies grader and acknowledges to student

regrade - Regrade Request Management

# List all open regrade requests
mlipcli regrade list

# List open regrade requests for a specific assignment
mlipcli regrade list I1

This lists all open regrade requests. Open regrade requests are submissions where the most recent comment starts with "Regrade request".

slack - Slack Operations

# Sync Slack user IDs for all students in roster
mlipcli slack sync

Maps Slack users to users on Canvas by email and reports mismatch. Mismatches can be manually fixed by adding the email of a Slack account in the Slack Email column of the roster spreadsheet for those students who did sign up with a different email.

breakout - Breakout Session Operations

Automated feedback for breakout session exercises using LLM analysis.

# Generate and send feedback (interactive mode)
mlipcli breakout feedback exercise1

# Dry run - show feedback without posting
mlipcli breakout feedback exercise1 --dry

# Test criteria against exported Slack messages
mlipcli breakout test exercise1 messages.json

See the breakout_config.yml file for the formatting of how to set up the autograder. Requires LLM access to be configured in .env.

Typical Workflow

1. Setup (once per semester):

   mlipcli roster init

2. Add/drop (when students join/drop)

   mlipcli roster update
   mlipcli slack sync

3. Per assignment:

   mlipcli assignment init I1
   mlipcli gradingassignment init I1

4. Monitoring (run continuously on some server, monitor with prometheus):

   mlipcli monitor start

5. Grading: Possible bulk autograding:

   # possible autograding: download all grades
   mlipcli grade download all I1
   ... run autograder to create grade.md files
   mlipcli upload I1

   Individual grading of assigned solutions:

   mlipcli grade download <ta-andrewid> I1   # Download assigned submissions
   # ... grade locally, fill in grade.md files ...
   mlipcli grade check I1            # Validate grades
   mlipcli grade upload I1           # Upload to spreadsheet
   # instructor
   mlipcli grade release I1          # Release to Canvas

6. Handle regrades: TAs are notified by Slack of incoming regrade requests. To update the grade use the standard grade upload I1 <andrewId> commands to upload a changed grade.md file. Either release the grade yourself grade release I1 <andrewId> (which will create a comment under your name) or ask the instructors to release an update on your behalf. If you do not want to change the grade, post a comment under the regrade request to close it (which is not anonymous unfortunately).

   Find all open regrade requests with

   mlipcli regrade list I1           # Check for regrade requests

Development

Prerequisites

• Deno 2.6+

Commands

# Type checking and linting
deno check
deno lint
deno fmt

# Build npm package (only for release to npm)
deno task build:npm

License

GPL-3.0