Skip to content

GroupBy.sort and sortBy vague error reporting #780

Open

Description

@koperagen

Given https://www.kaggle.com/datasets/ruchi798/data-science-job-salaries and

@DataSchema
interface DsSalaries {
    @ColumnName("company_location")
    val companyLocation: String
    @ColumnName("company_size")
    val companySize: String
    @ColumnName("employee_residence")
    val employeeResidence: String
    @ColumnName("employment_type")
    val employmentType: String
    @ColumnName("experience_level")
    val experienceLevel: String
    @ColumnName("job_title")
    val jobTitle: String
    @ColumnName("remote_ratio")
    val remoteRatio: Int
    val salary: Int
    @ColumnName("salary_currency")
    val salaryCurrency: String
    @ColumnName("salary_in_usd")
    val salaryInUsd: Int
    val untitled: Int
    @ColumnName("work_year")
    val workYear: Int
}

This doesn't fail

df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortBy(pathOf("group", "salaryInUsd")).print()

This does with "can't apply sort flag to column group":

df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortByDesc(pathOf("group", "salaryInUsd")).print()

In reality, both snippets are wrong and it should be sortBy(pathOf("group", "group", "salary_in_usd")) (which itself is confusing).
These functions should report that pathOf("group", "salaryInUsd") doesn't exist in both cases

======================================================================================

Problem

GroupBy.sortBy and GroupBy.sortByDesc handle an invalid nested path inconsistently.

Given a dataframe where salaryInUsd is grouped into a column group:

df.group { salaryInUsd }
    .into("group")
    .groupBy { companyLocation }

The following call does not fail, even though the path is incorrect:

.sortBy(pathOf("group", "salaryInUsd"))

But the descending version fails with a misleading error:

.sortByDesc(pathOf("group", "salaryInUsd"))

Current error:

can't apply sort flag to column group

In reality, both calls use a wrong path. The expected path is:

pathOf("group", "group", "salary_in_usd")

The current behavior is confusing because:

  • sortBy silently accepts an invalid path;
  • sortByDesc fails, but with an error that points to the wrong problem;
  • neither function clearly reports that pathOf("group", "salaryInUsd") does not exist.

Expected Behavior

Both GroupBy.sortBy and GroupBy.sortByDesc should validate the selected path consistently.

If the path does not exist, both functions should report a clear error, for example:

Column path 'group.salaryInUsd' does not exist in the grouped dataframe.

The error should preferably include enough context to help users understand the actual available nested path.

Definition of Done

  • Add regression tests for invalid nested paths in GroupBy.sortBy.
  • Add regression tests for invalid nested paths in GroupBy.sortByDesc.
  • Ensure both functions fail consistently for the same invalid path.
  • Replace the misleading can't apply sort flag to column group error with a clearer missing-path diagnostic when appropriate.
  • Verify that valid nested paths, such as pathOf("group", "group", "salary_in_usd"), still work.
  • Avoid changing sorting semantics beyond validation/error reporting.

Activity

  1. added this to the 0.14.0 milestone on Jul 19, 2024
  2. modified the milestones: 0.14.0, Backlog on Sep 16, 2024
  3. modified the milestones: Backlog, 1.0.0-Backlog on Apr 8, 2026
  4. modified the milestones: Post-RC, 1.1 on Jul 17, 2026
  5. zaleslaw commented on Jul 17, 2026

    @zaleslaw
    Collaborator

    Moving this to 1.0, pending verification on the current codebase.

    The important part here is not only the vague error message in sortByDesc, but the possibility that sortBy silently accepts a non-existing nested path. If confirmed, this is a correctness / fail-fast issue in a core workflow (groupBy + sorting), not just documentation polish.

    For 1.0, the scope should stay narrow: validate missing paths consistently in GroupBy.sortBy / sortByDesc, add regression tests, and improve the diagnostic without changing sorting semantics or public API.

  6. modified the milestones: 1.1, 1.0, Post-RC on Jul 17, 2026
  7. modified the milestones: Post-RC, 1.0 on Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions