mlipcli

A CLI tool for managing workflows in the Machine Learning in Production course across Canvas LMS, Google Sheets, and Slack. Handles student rosters, grading assignments, submission downloads, grade management, and automated monitoring.

Installation

Run the project with npx mlipcli <module> <command> [options] (Node.js 24+)

If you prefer to run from source, use deno: deno run --A src/main.ts

This project needs a number of configuration secrets, that can be set as environment variables or in a .env file in the working directory:

Canvas (for Canvas commands):

• CANVAS_API_URL - Canvas API URL (e.g., https://canvas.cmu.edu/api/v1)
• CANVAS_API_TOKEN - Canvas API token, created in the Canvas profile: https://canvas.cmu.edu/profile/settings
• CANVAS_COURSE_ID - Canvas course ID, the number in the URL of the course

Google Sheets (for spreadsheet commands):

• GOOGLE_CREDENTIALS_PATH - Path to Google service account credentials JSON
• SPREADSHEET_ID - id of the Google spreadsheet that stores various data (service account must have write access)
• GOOGLE_DRIVE_FOLDER_ID and SPREADSHEET_NAME - Alternative to SPREADSHEET_ID, identifying the spreadsheet by folder and name.

Slack (for Slack commands):

• SLACK_ACCESS_TOKEN - Slack bot token

LLM (for breakout feedback commands):

• LLM_API_KEY, LLM_MODEL, LLM_API_ADDRESS

Important Modules

grade - Grading Operations

Download Submissions.

# Download submissions assigned to a specific TA
mlipcli grade download I1 jdoe

# Download all submissions for an assignment
mlipcli grade download I1 all

Clones and checks out the repositories at the commit the students submitted. If the submission has been graded, it creates/overwrites the grade.md file. Submissions that are not graded yet get a blank grade.md from the assignment's rubric template (see assignment rubric below); an existing local grade.md is never overwritten this way.

Creates a directory structure:

grading/
  I1/
    .rubric_template.md     # Blank rubric of the assignment, cached for `grade ui`
    andrewId/               # Git repository cloned at submitted commit
      grade.md              # Rubric for grading
    andrewId_resubmission/  # If student resubmitted
      grade.md

Rubric format. Each rubric item has a title followed by checkbox options with points. Mark exactly one option with [x]. Other markings like [?] can be used for temporary markings that still need to be checked. Lines starting with // are treated as internal comments not released to students. When giving 0 points, an explanation after the colon is required.

The item's line is a short title, optionally followed by a semicolon and a longer description; with no semicolon the whole line is the title. The short title is what shows up in tables and lists, the description explains the requirement in full.

Options must start with * [ and the item's line must be directly above its options — a blank line in between makes the item invisible to the parser.

Installable; We can install and run your implementation based on the README, without undocumented steps.

- [x] 20p: Requirements met
- [ ] 0p: Requirements not met
      // file: README.md

No private credentials are committed to the repository.

- [ ] 10p: Requirements met
- [x] 0p: Requirements not met: API key found in config.js
      // file: src/**/*.py
      // internal comment, visible to TAs, but not released to students

// file: <path> lines name the submission files that are relevant for the rubric item above them; grade ui opens them automatically. Paths are relative to the submission and may use * and **.

The grade check command checks that the rubrics are filled out correctly for all downloaded solutions. The check command validates each rubric item:

• Exactly one option is checked (not zero, not multiple)
• If 0 points is checked, a comment explaining the deduction is provided
• No comments on unchecked options (likely a mistake)

# Validate all grade.md files are properly filled in
mlipcli grade check I1

The grade report provides a breakdown of grades across all submissions.

Grading interface. grade ui opens a local web interface for grading the submissions that were downloaded to grading/<assignment>/. It works entirely on the local files — no spreadsheet access — and writes every change straight to the corresponding grade.md.

# Open the grading interface for I1 (opens a browser at http://localhost:17646)
mlipcli grade ui I1

# Pick a different port, or do not open a browser
mlipcli grade ui I1 --port=8080 --no-open

The interface shows the rubric and the student's solution side by side and supports two ways of working:

• By student — the full rubric for one submission at a time, the usual workflow.
• By question — one rubric item pinned while stepping through all students, the way Gradescope works. The relevant file of that item (from its // file: hints) is opened for each student automatically, and comments already used for the same rubric option are offered for reuse with one click.

Both the student-facing comments and the internal // notes are editable, the whole submission folder can be browsed, rubric problems are flagged as you grade, and rubrics that drifted from the assignment template are called out. The submission list can be sorted by name or by points. Nothing is sent anywhere: run grade upload when you are done.

The Statistics page breaks the assignment down by rubric item, showing what share of submissions is done, ungraded, needing review, or invalid for each; picking a number lists those submissions and opens them at that rubric item.

What is on screen is reflected in the URL, so a specific submission and rubric item can be linked to or bookmarked: /jdoe/all for a whole submission, /jdoe/2 for its second rubric item, /stats for the statistics page. The browser's back and forward buttons work.

Keyboard: j/k next or previous submission, ⇧J/⇧K next or previous that is not done, n/p next or previous rubric item, 1–9 pick an option, c jump to the comment, ? help.

If you edit a grade.md in an editor while the interface is open, the change is detected and reloaded rather than overwritten.

Upload Grades.

Upload all grades from local grade.md files to the spreadsheet. This does not automatically release the grades.

# Upload grades from local grade.md files to spreadsheet
mlipcli grade upload I1

# Upload with automatic commit hash insertion
mlipcli grade upload I1 --addHash

# Upload specific students only
mlipcli grade upload I1 student1 student2

Release Grades. Instructors only: Instructors can check and release grades, which posts them to Canvas (after possible point penalties). Note that the grades are posted with a comment that is associated with the account pushing the grades.

# Release grades from spreadsheet to Canvas with comments
mlipcli grade release I1

# Release specific students only
mlipcli grade release I1 student1 student2

Releasing grades:

• Posts the grade to Canvas
• Posts a comment with the rubric
• Sends a Slack notification to the student

roster - Course Roster Operations

Manage student roster synchronization between Canvas and Google Sheets.

# Update roster from Canvas (marks removed students, adds new ones)
mlipcli roster update

# List all non-dropped students and TAs
mlipcli roster list

assignment - Assignment Setup

# Create the spreadsheet columns for an assignment
mlipcli assignment init I1

# Store the blank rubric of an assignment (validated before it is stored)
mlipcli assignment rubric I1 I1_rubric.md

# Print the rubric template currently stored for an assignment
mlipcli assignment rubric I1

The rubric template is the blank rubric that grade download writes for submissions that are not graded yet, and that grade ui checks each submission against. It is stored per assignment on the Rubrics sheet of the spreadsheet, so every TA grades against the same rubric. Put the // file: hints for the grading interface into the template.

gradingassignment - Grading Assignment Operations

Assign TAs to grade students for specific assignments.

# Create grading assignment column and assign TAs to students
mlipcli gradingassignment init I1

The algorithm assigns TAs to students while:

• Balancing workload across TAs
• Avoiding conflicts (defined in spreadsheet)
• Trying not to assign the same TA to a student repeatedly

monitor - Submission Monitoring

Instructor only: Long-running process that monitors Canvas for new submissions and comments and updates the spreadsheet with the latest submission URLs (from where the grade command can download them). This also handles late days, token use, and regrade requests.

# Start continuous monitoring (checks every 10 minutes)
mlipcli monitor start

# Run a single check without processing (dry run)
mlipcli monitor test

# Process all pending items once and exit
mlipcli monitor once

The monitor handles:

• New submissions: Records submission URL, handles late tokens
• Resubmissions: Validates tokens, notifies graders
• Invalid URLs: Notifies students via Slack and Canvas comment
• Regrade requests: Notifies grader and acknowledges to student

regrade - Regrade Request Management

# List all open regrade requests
mlipcli regrade list

# List open regrade requests for a specific assignment
mlipcli regrade list I1

This lists all open regrade requests. Open regrade requests are submissions where the most recent comment starts with "Regrade request".

slack - Slack Operations

# Sync Slack user IDs for all students in roster
mlipcli slack sync

Maps Slack users to users on Canvas by email and reports mismatch. Mismatches can be manually fixed by adding the email of a Slack account in the Slack Email column of the roster spreadsheet for those students who did sign up with a different email.

participation - Lecture Participation Tracking

Track lecture participation from Slack #lecture and #social channels. Participation is recorded on a separate "Participation" sheet in the same spreadsheet.

# Initialize participation sheet with lecture dates
mlipcli participation init --start 2025-01-13 --end 2025-04-23 --days mo,we --time 18

# Print Andrew IDs credited in a custom lecture window
mlipcli participation collect 2025-01-13 18 20
mlipcli participation collect 2025-01-13 1805 18:55

# Update participation from Slack
mlipcli participation update

# One-time end-of-semester token application to participation
mlipcli participation applytokens D O 12345 67890

The init command creates the Participation sheet with columns for each lecture date and populates it with the current student roster. The update command searches Slack for participation posts:

• #lecture: Messages during the lecture time window (start time + 2 hours). If a student tags other students, all involved get "g" (group). Solo posts get "s". "g" takes priority over "s".
• #social: With --intros, posts in the first 15 days after the first lecture credit their author as "g" in the first lecture column.

Update is incremental — it only processes date columns that have no data yet; --all reprocesses every past date and overwrites what is there.

A cell may hold "s" (solo, worth nothing), "g" (answered with at least one partner), "f" (a "g" whose answer met every criterion or whose feedback was followed up), or "excused". Both "g" and "f" count as one participation in postgrade and postweighted; "excused" is left out of the total.

The Slack bot records participation as it happens. While deno task slackbot is running it writes "g" and "f" for every answer it gives feedback on in #lecture, so participation update is only needed to backfill a lecture the bot missed — and since it skips date columns that already have data, that means participation update --all. See src/slackbot/README.md.

applytokens is intended for one-time semester-end use. It checks opted-in students, uses the minimum token count needed to achieve the best participation-grade improvement, updates the remaining-token Canvas assignment with a comment, and overwrites the tokensused column in the Participation sheet for this run. After it completes, rerun participation postgrade to publish updated participation grades.

breakout - Breakout Session Operations

Automated feedback for breakout session exercises using LLM analysis.

# Generate and send feedback (interactive mode)
mlipcli breakout feedback exercise1

# Dry run - show feedback without posting
mlipcli breakout feedback exercise1 --dry

# Test criteria against exported Slack messages
mlipcli breakout test exercise1 messages.json

See the breakout_config.yml file for the formatting of how to set up the autograder. Requires LLM access to be configured in .env.

Slack Bot

A long-running bot that gives students feedback on breakout answers during class, as an alternative to running mlipcli breakout feedback afterwards. It is a separate process from the CLI and is not part of the published npm package.

deno task slackbot:demo  # see both feedback mechanisms offline -- no Slack, tokens, or LLM
deno task slackbot       # run the bot (needs a Slack app and slackbot_config.yml)
deno task test           # run the tests

Two ways a student gets feedback: tagging the bot in the lecture channel, which replies in the message thread where the class can see it, or /breakout <answer>, which only they see. Students also have /participation to opt in or out of participation grading, and /grading-status to see their open regrade requests and which of their submissions have been graded.

Students also create their own assignment repositories with /create-repo <assignment> <github-username>, replacing GitHub Classroom: a private repository in the course organization, copied from a template, with the student invited and the staff team given access.

/onlinelab join puts a student in the queue for the weekly online lab, /onlinelab cancel gives their slot back, and /onlinelab status says where they stand. The queue is a Google spreadsheet of its own, one tab per lab day, which the TAs tick off as they work down it.

Full documentation, configuration, and roadmap: src/slackbot/README.md. Slack app setup: src/slackbot/doc/slack-app-setup.md.

Typical Workflow

1. Setup (once per semester):

   mlipcli roster init

2. Add/drop (when students join/drop)

   mlipcli roster update
   mlipcli slack sync

3. Per assignment:

   mlipcli assignment init I1
   mlipcli assignment rubric I1 I1_rubric.md   # store the blank rubric
   mlipcli gradingassignment init I1

4. Monitoring (run continuously on some server, monitor with prometheus):

   mlipcli monitor start

5. Grading: Possible bulk autograding:

   # possible autograding: download all grades
   mlipcli grade download I1 all
   ... run autograder to create grade.md files
   mlipcli grade upload I1

   Individual grading of assigned solutions:

   mlipcli grade download I1 <ta-andrewid>   # Download assigned submissions
   mlipcli grade ui I1               # Grade in the browser (or edit grade.md by hand)
   mlipcli grade check I1            # Validate grades
   mlipcli grade upload I1           # Upload to spreadsheet
   # instructor
   mlipcli grade release I1          # Release to Canvas

6. Handle regrades: TAs are notified by Slack of incoming regrade requests. To update the grade use the standard grade upload I1 <andrewId> commands to upload a changed grade.md file. Either release the grade yourself grade release I1 <andrewId> (which will create a comment under your name) or ask the instructors to release an update on your behalf. If you do not want to change the grade, post a comment under the regrade request to close it (which is not anonymous unfortunately).

   Find all open regrade requests with

   mlipcli regrade list I1           # Check for regrade requests

Development

Prerequisites

• Deno 2.6+

Commands

# Run from source
deno task run <module> <command> [options]

# Type checking, linting, formatting and tests
deno check src/main.ts src/gradingui/web/index.tsx
deno lint
deno fmt
deno task test

# Rebuild the grading UI bundle after changing anything below src/gradingui/web
deno task build:ui

# Build npm package (only for release to npm; runs build:ui first)
deno task build:npm

Grading UI

The browser app for grade ui lives in src/gradingui/web (React + Bulma) and talks to the small HTTP server in src/modules/grade.ui.ts. deno task build:ui bundles it with esbuild into src/gradingui/assets.generated.ts, which is committed so that the CLI runs from a fresh clone and from the npm package without a build step. Rerun the task and commit the result whenever you change the app.

License

GPL-3.0