메인 콘텐츠로 건너뛰기
weave / Evaluation scorers 세트와 Datasets 를 포함하는 evaluation을 설정합니다. evaluation.evaluate(model)을 호출하면 Datasets 의 각 행이 모델로 전달되며, 이때 Datasets 의 컬럼 이름과 model.predict의 인수(argument) 이름을 매칭합니다. 그 후 모든 scorers를 호출하고 결과를 Weave 에 저장합니다. 예시
// 예시 데이터들을 모아 Datasets 생성
const dataset = new weave.Dataset({
  id: 'my-dataset',
  rows: [
    { question: 'What is the capital of France?', expected: 'Paris' },
    { question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee' },
    { question: 'What is the square root of 64?', expected: '8' },
  ],
});

// 커스텀 scoring 함수 정의
const scoringFunction = weave.op(function isEqual({ modelOutput, datasetRow }) {
  return modelOutput == datasetRow.expected;
});

// 평가할 함수 정의
const model = weave.op(async function alwaysParisModel({ question }) {
  return 'Paris';
});

// 평가 시작
const evaluation = new weave.Evaluation({
  id: 'my-evaluation',
  dataset: dataset,
  scorers: [scoringFunction],
});

const results = await evaluation.evaluate({ model });

Type parameters

NameType
Rextends DatasetRow
Eextends DatasetRow
MM

Hierarchy

목차

생성자 (Constructors)

속성 (Properties)

Accessors

메소드 (Methods)

생성자 (Constructors)

constructor

new Evaluation<R, E, M>(parameters): Evaluation<R, E, M>

Type parameters

NameType
Rextends DatasetRow
Eextends DatasetRow
MM

파라미터 (Parameters)

NameType
parametersEvaluationParameters<R, E, M>

반환값 (Returns)

Evaluation<R, E, M>

Overrides

WeaveObject.constructor

Defined in

evaluation.ts:148

속성 (Properties)

__savedRef

Optional __savedRef: ObjectRef | Promise<ObjectRef>

Inherited from

WeaveObject.__savedRef

Defined in

weaveObject.ts:73

Accessors

description

get description(): undefined | string

반환값 (Returns)

undefined | string

Inherited from

WeaveObject.description

Defined in

weaveObject.ts:100

name

get name(): string

반환값 (Returns)

string

Inherited from

WeaveObject.name

Defined in

weaveObject.ts:96

메소드 (Methods)

evaluate

evaluate(«destructured»): Promise<Record<string, any>>

파라미터 (Parameters)

NameTypeDefault value
«destructured»Objectundefined
› maxConcurrency?number5
› modelWeaveCallable<(…args: [{ datasetRow: R }]) => Promise<M>>undefined
› nTrials?number1

반환값 (Returns)

Promise<Record<string, any>>

Defined in

evaluation.ts:163

predictAndScore

predictAndScore(«destructured»): Promise<{ model_latency: number = modelLatency; model_output: any = modelOutput; model_success: boolean = !modelError; scores: { [key: string]: any; } }>

파라미터 (Parameters)

NameType
«destructured»Object
› columnMapping?ColumnMapping<R, E>
› exampleR
› modelWeaveCallable<(…args: [{ datasetRow: E }]) => Promise<M>>

반환값 (Returns)

Promise<{ model_latency: number = modelLatency; model_output: any = modelOutput; model_success: boolean = !modelError; scores: { [key: string]: any; } }>

Defined in

evaluation.ts:231

saveAttrs

saveAttrs(): Object

반환값 (Returns)

Object

Inherited from

WeaveObject.saveAttrs

Defined in

weaveObject.ts:77