Repository navigation
GroupBy.sort and sortBy vague error reporting #780
Description
Given https://www.kaggle.com/datasets/ruchi798/data-science-job-salaries and
@DataSchema
interface DsSalaries {
@ColumnName("company_location")
val companyLocation: String
@ColumnName("company_size")
val companySize: String
@ColumnName("employee_residence")
val employeeResidence: String
@ColumnName("employment_type")
val employmentType: String
@ColumnName("experience_level")
val experienceLevel: String
@ColumnName("job_title")
val jobTitle: String
@ColumnName("remote_ratio")
val remoteRatio: Int
val salary: Int
@ColumnName("salary_currency")
val salaryCurrency: String
@ColumnName("salary_in_usd")
val salaryInUsd: Int
val untitled: Int
@ColumnName("work_year")
val workYear: Int
}This doesn't fail
df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortBy(pathOf("group", "salaryInUsd")).print()This does with "can't apply sort flag to column group":
df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortByDesc(pathOf("group", "salaryInUsd")).print()In reality, both snippets are wrong and it should be sortBy(pathOf("group", "group", "salary_in_usd")) (which itself is confusing).
These functions should report that pathOf("group", "salaryInUsd") doesn't exist in both cases
======================================================================================
Problem
GroupBy.sortBy and GroupBy.sortByDesc handle an invalid nested path inconsistently.
Given a dataframe where salaryInUsd is grouped into a column group:
df.group { salaryInUsd }
.into("group")
.groupBy { companyLocation }The following call does not fail, even though the path is incorrect:
.sortBy(pathOf("group", "salaryInUsd"))But the descending version fails with a misleading error:
.sortByDesc(pathOf("group", "salaryInUsd"))Current error:
can't apply sort flag to column group
In reality, both calls use a wrong path. The expected path is:
pathOf("group", "group", "salary_in_usd")The current behavior is confusing because:
sortBysilently accepts an invalid path;sortByDescfails, but with an error that points to the wrong problem;- neither function clearly reports that
pathOf("group", "salaryInUsd")does not exist.
Expected Behavior
Both GroupBy.sortBy and GroupBy.sortByDesc should validate the selected path consistently.
If the path does not exist, both functions should report a clear error, for example:
Column path 'group.salaryInUsd' does not exist in the grouped dataframe.
The error should preferably include enough context to help users understand the actual available nested path.
Definition of Done
- Add regression tests for invalid nested paths in
GroupBy.sortBy. - Add regression tests for invalid nested paths in
GroupBy.sortByDesc. - Ensure both functions fail consistently for the same invalid path.
- Replace the misleading
can't apply sort flag to column grouperror with a clearer missing-path diagnostic when appropriate. - Verify that valid nested paths, such as
pathOf("group", "group", "salary_in_usd"), still work. - Avoid changing sorting semantics beyond validation/error reporting.
Activity
zaleslaw commentedon Jul 17, 2026
Moving this to 1.0, pending verification on the current codebase.
The important part here is not only the vague error message in
sortByDesc, but the possibility thatsortBysilently accepts a non-existing nested path. If confirmed, this is a correctness / fail-fast issue in a core workflow (groupBy+ sorting), not just documentation polish.For 1.0, the scope should stay narrow: validate missing paths consistently in
GroupBy.sortBy/sortByDesc, add regression tests, and improve the diagnostic without changing sorting semantics or public API.