Repository navigation
Update readme - #1706
Update readme#1706
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
bkorycki
left a comment
bkorycki
left a comment
There was a problem hiding this comment.
Looks good! There could be more code to rip out if we think that default evaluators are gone for good. For example, we aren't using the public safe tests anymore. So that object hierarchy could be simplified. Not strictly necessary for this PR but something to keep in mind for possible future clean up efforts.
| you will need accounts with those services as well. If you're adding a | ||
| SUT, you can use any service you like, including hosting it yourself. | ||
| Benchmarks evaluate the safety of responses using MLCommons' | ||
| ensemble evaluator (the `cheval` service). You will need credentials |
There was a problem hiding this comment.
Nitpicky but not all of our evaluators are ensembles anymore.
There was a problem hiding this comment.
Not nitpicky, I'll fix it! Thanks!
|
@bkorycki , when I talked with Bennet he said we can rip it out and then revisit in Nov or so. So, I didn't want to undo a lot of plumbing until we've had those discussions. But you're right, let's definitely circle back when we have a definitive direction for the future and decide if we want to take it out for good, or figure out some sustainable way to keep it included going forward. |
GitHub Actions
Failure
| SUT, you can use any service you like, including hosting it yourself. | ||
| Benchmarks evaluate the safety of responses using MLCommons' | ||
| ensemble evaluator (the `cheval` service). You will need credentials | ||
| for it -- ask the MLCommons admin for permission. The example SUT |
There was a problem hiding this comment.
This seems like the correct thing to say for the launch, but I don't think this makes sense long term. We should figure out a clear strategy here, but that's mainly a business decision.
There was a problem hiding this comment.
I agree. I'm aiming for correctness now and we can make it correct for a better situation later.
And remove --evaluator option, defaulting to "private"