Skip to content

fix: improve genstubs and some related bugs - #1781

Open
jakelorocco wants to merge 2 commits into
generative-computing:mainfrom
jakelorocco:fix/genstubs
Open

jakelorocco wants to merge 2 commits into
generative-computing:mainfrom
jakelorocco:fix/genstubs

Conversation

@jakelorocco

Copy link
Copy Markdown
Contributor

Pull Request

Issue

Fixes #1476

Description

Improves genstub performance by modifying the template. On set of genstub tasks based on our examples and other scenarios, performance improved by 3.1% averaged across models and sizes. Granite models saw large improvements along with models with thinking on.

Performance saw large improvements when including the return format in the prompt.

Additional bug fixes:

  1. -> X | None stubs crashed when the right answer was None. When the model correctly returned {"result": null}, a check in mellea/core/base.py required every result to be non-None and raised an AssertionError.
  2. Template lookup
  3. Prompt rendering fixes
    a. docstrings left out when empty instead of rendering as """"""
    b. argument types fixed
    c. removed module paths from types
    d. pydantic arguments are rendered as json along with dicts and lists
    e. unquoted unresolved return types to prevent small models from returning strings
    f. multiline docstrings keep their indentation

Testing

  • Tests added to the respective file if code was changed
  • New code has 100% coverage if code was added
  • Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

  • AI coding assistants used

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

  • Component
  • Requirement
  • Sampling Strategy
  • Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com>
Assisted-by: CLAUDE:OPUS
@github-actions github-actions Bot added the bug Something isn't working label Oct 30, 2026
Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com>
Assisted-by: CLAUDE:OPUS
@jakelorocco
jakelorocco marked this pull request as ready for review November 3, 2026 20:22
@jakelorocco
jakelorocco requested a review from a team as a code owner November 3, 2026 20:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generative stub prompt template underperforms on smaller models

1 participant