Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
This document summarizes notable updates since February 2025, with commit dates from the repository history for context.【728428†L1-L48】

## 2026-07
- **Adaptation (2.1.9):** English-to-LTL questions were structurally rare, and each of their three framing arms rarer still. Question selection treated `tracesatisfaction_mc`, `tracesatisfaction_yn` and `englishtoltl` as three peers and split the probability mass between them, so the trace-reading skill got two shares to english-to-LTL's one: 2/3 of draws with no history at all, and up to 85% once the 0.15 per-type exploration floor bound the rest — leaving english-to-LTL at 5% per framing arm, i.e. one deontic question in twenty. Selection is now hierarchical over the two skills actually being practised. `QUESTION_FAMILIES` groups the types into a trace-satisfaction family (mc, yn) and an english-to-LTL family; `calculate_question_family_weights` scores a family on its *pooled* record with the same Laplace-smoothed error rate as before and a 0.3 floor, and `calculate_question_type_weights` splits each family's weight evenly across its subtypes, so it keeps returning a distribution over the three types and the profile page and JSON export are unchanged in shape. Cold start is now 50% english-to-LTL / 25% each trace type rather than 33/33/33, and a student who has mastered english-to-LTL still sees it 30% of the time rather than 15%. Measured end to end over 40 generated exercises with real SPOT: 52% english-to-LTL (was 40%), with every exercise containing at least one (7.5% contained none before). Subtypes are deliberately not drilled against each other — a yes/no trace question is guessable at 50% and a multiple-choice one at ~17%, so their raw error rates were never comparable, and both read a trace against a formula either way. The three english-to-LTL framings stay uniformly assigned per question, since they are randomized experiment arms; they now simply accrue faster (each ~17% of questions at cold start, floored at 10%). Analyses spanning this change should segment on it: the per-arm *ratio* is untouched, but per-student exposure counts shift.
- **UX (2.1.9):** A generated exercise is now 4 to 6 questions rather than 3 to 8. An exercise is a sitting, and a range that wide made the length of that sitting unknowable — two exercises in a row could differ by more than a factor of two.
- **UX (2.1.9):** The profile page's "Question Mix" card now shows the mix by skill (Trace satisfaction, English to LTL) rather than by question type. Selection adapts per family and splits a family's weight evenly across its types, so the two trace-satisfaction bars would have been equal at every point in every student's history — three bars where two never differ reads as a rendering bug. Its caption ("Types you miss more are drilled more") was also false after the family split, since missing yes/no questions raises both trace bars equally rather than drilling yes/no; it now reads "Skills you miss more are drilled more." The per-type weights are still what a question is drawn with, so they stay in the JSON export, which gains `question_family_weights` alongside them and goes to `schema_version` 3.
- **UX (2.1.9):** The "Estimated Misconceptions Over Time" chart on the profile page is collapsed by default. It is the most detailed thing on the page and the least glanceable, so the page now opens on the summary cards. The heading is a real `<button>` inside its `<h3>` (focusable and operable from the keyboard, unlike the `<p role="button">` disclosures elsewhere, which have no `tabindex`), carrying the same rotating `.toggle-arrow` triangle used on the login page — an affordance that does not depend on color. Bootstrap's `.btn:focus` glow is suppressed on it: being `:focus` rather than `:focus-visible`, it stayed painted around the heading after a mouse click, in Bootstrap's own default blue rather than the theme's, and the `:focus-visible` ring for keyboard users is restated at (0,3,0) to outrank `.btn:focus { outline: 0 }`. Chart.js measures its canvas as it constructs the chart, and a canvas inside a collapsed container measures zero, so the chart is built the first time the section is opened rather than on page load.
- **Adaptation (2.1.9):** A generated exercise now reserves its first slots for the highest-scoring question of each *family* rather than of each type. `num_questions` is drawn from [3, 8], so at the low end reserving one slot per type was the entire exercise: a 3-question exercise was always exactly one of each, and the selection weights chose nothing at all. Both skills still always appear.
- **UX (2.1.8):** Stepping through a trace no longer redraws the page. Every trace state used to be rendered as its own copy of the formula tree, and a step cross-faded between two of them: for 250ms both copies were `display: block` inside a `flex-wrap` container, so the stepper occupied two rows and everything below it — the rule, the "different trace" button, the footer — was shoved down and snapped back on every click. The incoming copy also rebuilt its trace diagram from scratch (`TraceRenderer.render` clears the container) even though the diagram is identical at every step bar the highlight, and a 1s `flashHighlight` animation fired over the tree each time. Between the three, a step read as a page reload. Only two things actually differ between steps, so only those change now: the `tree-sat`/`tree-unsat` class on each node of a single tree, and which state carries the highlight. `TraceSatisfactionResult.getStepperViewData` sends the tree once plus one truth vector per state, addressed by a new `data-node-index` emitted in the same pre-order `getAllSubformulae` walks; `TraceRenderer.setHighlight` moves the highlight by toggling attributes on the existing SVG (every state gets a `CURRENT` badge at render time, hidden but for the current one), so nothing is created, measured or resized. Verified in the running app: document height, button position and the SVG element itself are unchanged across all steps, with no network requests. The 250ms transition also gated input — presses inside the window were dropped, so holding an arrow key lost most of them — and that guard is gone, so stepping keeps up with the keyboard. The empty stepper tab now says so, matching the Table and Matrix tabs, and the arrow buttons carry `aria-label`s rather than relying on `title` over an icon.
- **UX (2.1.8):** Removed the "No further feedback is currently available. We recommend stepping through the trace to see where/if it diverges from the formula." message from trace-satisfaction yes/no questions. It was the fallback for the half of those questions whose trace genuinely satisfies the formula (`TRACESAT_YES_PROBABILITY = 0.5`), where there is no misconception formula to contrast against — but it has been stale since per-state feedback landed in 2.1.6: it printed *below* a paragraph that already explains the verdict ("This trace **does** satisfy the formula… a trace satisfies the formula exactly when it holds from the very first state"), and its step-through recommendation duplicated the "Want to see it step by step?" stepper button below it. Such questions now carry no predetermined feedback (`feedback` is `None`; both templates already guarded on it). The "No" case keeps its contrastive feedback, which names the formula the trace *does* satisfy — something the per-state marks don't say. If the per-state block can't render (SPOT evaluation fails, or the state count doesn't match the drawn trace), `displayTraceSatFeedback` still returns silently, so those rare cases now show the verdict and the stepper button alone.
- **Bugfix (2.1.6):** The trace diagrams inside misconception explainers never rendered. Explainers are shown in the "recommended review" card on the home page, but `index.html` was the one page emitting `.trace-diagram` markup without loading `tracerenderer.js`, so the diagrams stayed empty `<div>`s — leaving the Finally explainer ending on "Thus both these traces satisfy `F p`:" followed by nothing, and the Until explainer missing the trace its closing sentence refers to. It looked intermittent because only two of the eight explainers (`implicitf`, `exclusiveu`) illustrate the operator with traces, and the card shows one explainer drawn at random from the student's top two misconceptions. The home page now loads the renderer and runs a render pass. The three copies of that pass (home page, exercise questions, answer feedback) are now one `TraceRenderer.renderAll(root)` helper, so a page that emits trace markup only has to call it.
Expand Down
4 changes: 3 additions & 1 deletion src/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -1118,7 +1118,9 @@ def generate_new_name():
else:
num_literals = random.randint(2, 4)
LITERALS = random.sample(literals_pool, num_literals)
num_questions = random.randint(3, 8)
## Kept narrow on purpose: an exercise is a sitting, and one that can be
## anywhere from 3 to 8 questions makes that sitting an unknown length.
num_questions = random.randint(4, 6)

data = exercise_builder.build_exercise(literals = LITERALS, num_questions = num_questions)
data = exerciseprocessor.randomize_questions(data)
Expand Down
134 changes: 96 additions & 38 deletions src/exercisebuilder.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,27 @@ class ExerciseBuilder:
ENGLISHTOLTL = "englishtoltl"
QUESTION_TYPES = [TRACESATMC, TRACESATYN, ENGLISHTOLTL]

## No question type's selection probability ever drops below this
## (exploration floor), no matter how well the student does on it.
QUESTION_TYPE_FLOOR = 0.15
## The question types group into two families, one per skill practised:
## reading a trace against a formula, and formalizing English into LTL.
## What sits below a family is a presentation variant of the same skill
## (multiple-choice vs yes/no; and, inside english-to-LTL, the abstract /
## lights / abac framings chosen in build_exercise). So selection is
## hierarchical: adapt between families, then draw a variant uniformly.
## Treating the two trace variants as peers of english-to-LTL instead gave
## the trace skill two shares of the mass to english-to-LTL's one -- 2/3 of
## questions with no history at all, and up to 85% once the floor bound the
## rest, which left each english-to-LTL framing at 5% of what a student saw.
TRACESAT_FAMILY = "tracesatisfaction"
ENGLISHTOLTL_FAMILY = "englishtoltl_family"
QUESTION_FAMILIES = {
TRACESAT_FAMILY: [TRACESATMC, TRACESATYN],
ENGLISHTOLTL_FAMILY: [ENGLISHTOLTL],
}

## No family's selection probability ever drops below this (exploration
## floor), no matter how well the student does on it. With two families
## this bounds adaptation to a 30/70 swing either way.
QUESTION_FAMILY_FLOOR = 0.3

# Trace yes/no questions first choose a misconception to target, then
# independently choose whether to show a positive or diagnostic-negative
Expand Down Expand Up @@ -60,6 +78,7 @@ def __init__(self, userLogs, complexity=5, syntax="Classic", misconception_oppor
self.syntax = syntax

self._distinct_answers_cache = None
self._question_family_weights_cache = None
self._question_type_weights_cache = None
self._misconception_weights_cache = None

Expand Down Expand Up @@ -170,54 +189,90 @@ def _distinct_answers(self):
self._distinct_answers_cache = answers
return answers

def calculate_question_type_weights(self):
@classmethod
def family_of(cls, question_type):
"""The family a question type belongs to, or None if unrecognized."""
for family, subtypes in cls.QUESTION_FAMILIES.items():
if question_type in subtypes:
return family
return None

def calculate_question_family_weights(self):
"""
Selection weights per question type, biased toward the types the
student gets wrong. Uses a Laplace-smoothed error rate
(incorrect + 1) / (attempts + 2), so with no history every type gets
0.5 and selection is uniform. After normalization, a floor guarantees
every type keeps at least QUESTION_TYPE_FLOOR probability.
Selection weights per question family, biased toward the family the
student gets wrong. Attempts are pooled across a family's subtypes,
then scored by a Laplace-smoothed error rate
(incorrect + 1) / (attempts + 2), so with no history every family gets
0.5 and selection is an even split. After normalization, a floor
guarantees every family keeps at least QUESTION_FAMILY_FLOOR
probability.

Pooling rather than scoring each subtype separately is deliberate: a
yes/no trace question is guessable at 50% and a multiple-choice one at
~17%, so their raw error rates are not comparable, and reading a trace
is one skill either way.
"""
if self._question_type_weights_cache is not None:
return self._question_type_weights_cache
if self._question_family_weights_cache is not None:
return self._question_family_weights_cache

counts = {qtype: {"attempts": 0, "incorrect": 0} for qtype in self.QUESTION_TYPES}
counts = {family: {"attempts": 0, "incorrect": 0} for family in self.QUESTION_FAMILIES}
for answer in self._distinct_answers():
qtype = getattr(answer, 'question_type', None)
if qtype not in counts:
family = self.family_of(getattr(answer, 'question_type', None))
if family is None:
continue
counts[qtype]["attempts"] += 1
counts[family]["attempts"] += 1
if not self._log_is_correct(answer):
counts[qtype]["incorrect"] += 1
counts[family]["incorrect"] += 1

error_rates = {
qtype: (c["incorrect"] + 1) / (c["attempts"] + 2)
for qtype, c in counts.items()
family: (c["incorrect"] + 1) / (c["attempts"] + 2)
for family, c in counts.items()
}
total = sum(error_rates.values())
normalized = {qtype: rate / total for qtype, rate in error_rates.items()}
normalized = {family: rate / total for family, rate in error_rates.items()}

## Apply the exploration floor: floored types get exactly the floor and
## the remaining mass is split proportionally among the rest. Repeat in
## case the rescaling pushes another type under the floor.
## Apply the exploration floor: floored families get exactly the floor
## and the remaining mass is split proportionally among the rest.
## Repeat in case the rescaling pushes another family under the floor.
result = dict(normalized)
floored = set()
for _ in range(len(result)):
low = {k for k, v in result.items() if k not in floored and v < self.QUESTION_TYPE_FLOOR}
low = {k for k, v in result.items() if k not in floored and v < self.QUESTION_FAMILY_FLOOR}
if not low:
break
floored |= low
free_keys = [k for k in result if k not in floored]
free_mass = 1.0 - self.QUESTION_TYPE_FLOOR * len(floored)
free_mass = 1.0 - self.QUESTION_FAMILY_FLOOR * len(floored)
free_total = sum(normalized[k] for k in free_keys)
for k in floored:
result[k] = self.QUESTION_TYPE_FLOOR
result[k] = self.QUESTION_FAMILY_FLOOR
for k in free_keys:
if free_total > 0:
result[k] = normalized[k] * free_mass / free_total
else:
result[k] = free_mass / len(free_keys)

self._question_family_weights_cache = result
return result

def calculate_question_type_weights(self):
"""
Per-type selection probabilities: each family's weight split evenly
across its subtypes, since a subtype is a presentation variant of the
same skill rather than a thing to drill separately. Drawing a type
from these marginals is the same as picking a family and then a
variant within it.
"""
if self._question_type_weights_cache is not None:
return self._question_type_weights_cache

family_weights = self.calculate_question_family_weights()
result = {
qtype: family_weights[family] / len(subtypes)
for family, subtypes in self.QUESTION_FAMILIES.items()
for qtype in subtypes
}

self._question_type_weights_cache = result
return result

Expand Down Expand Up @@ -433,19 +488,18 @@ def formula_choice_metric(formula):
# sort questions by score
chosen_questions = sorted(questions, key=lambda x: x['score'], reverse=True)

# Now choose the question with the highest metric, that is of each type from the chosen_questions
highest_ltl_to_eng = next((q for q in chosen_questions if q['type'] == self.ENGLISHTOLTL), None)
highest_trace_sat_mc = next((q for q in chosen_questions if q['type'] == self.TRACESATMC), None)
highest_trace_sat_yn = next((q for q in chosen_questions if q['type'] == self.TRACESATYN), None)


# Seed the exercise with the highest-scoring question of each *family*,
# so both skills are always practised, and fill the rest by score
# alone. Reserving a slot per subtype instead pinned every short
# exercise to one-of-each: callers ask for a handful of questions, so
# that reservation was most of the exercise and the selection weights
# were left choosing almost none of it.
final_choices = []
if highest_ltl_to_eng is not None:
final_choices.append(highest_ltl_to_eng)
if highest_trace_sat_mc is not None:
final_choices.append(highest_trace_sat_mc)
if highest_trace_sat_yn is not None:
final_choices.append(highest_trace_sat_yn)
for family in self.QUESTION_FAMILIES:
best = next((q for q in chosen_questions
if self.family_of(q['type']) == family), None)
if best is not None:
final_choices.append(best)

remaining = num_questions - len(final_choices)
if remaining > 0:
Expand Down Expand Up @@ -915,7 +969,10 @@ def get_profile_snapshot(self):
- complexity: the current difficulty level and its band, plus bounds
- misconception_snapshot: per-misconception weights the distractor
sampler uses now (the enum prefix stripped), most-likely first
- question_type_weights: the selection weights per question type
- question_family_weights: the selection weights per skill, which is
the level selection actually adapts at
- question_type_weights: those weights split evenly across each
family's presentation variants

Pure w.r.t. the builder's logs (no SPOT, no DB), so it is safe to call
from a request handler and straightforward to unit-test."""
Expand All @@ -932,6 +989,7 @@ def get_profile_snapshot(self):
"complexity_max": self.COMPLEXITY_MAX,
"complexity_band": self._complexity_band(),
"misconception_snapshot": misconception_snapshot,
"question_family_weights": self.calculate_question_family_weights(),
"question_type_weights": self.calculate_question_type_weights(),
}

Expand Down
Loading
Loading