ai training companies role scorecard template

What should a busy buyer put in the scorecard?

Write down what the person must deliver, how you will test it, and what would rule out a hire. Do this before reviewing resumes. Otherwise, each interview can become a test of whatever that interviewer happens to value.

Start by defining "AI training." You might need someone to label data, compare model responses, build evaluation tools, or teach employees to use AI. Those jobs need different evidence. A strong workshop instructor may have little experience judging model outputs against a detailed rubric.

For a role that spans several tasks, name the primary job. State which duties are occasional and which require independent work from the start. If the hire will own both technical systems and quality review, test both responsibilities.

Use the template below as a hiring document. Replace each bracketed field before sending it to a provider. The example weights and decision rules are proposed starting points, not industry benchmarks.

Copy the role definition

Avoid requirements such as "AI expert" or "excellent communicator." Write observable tasks instead. For example: "Explains why a model response fails the policy and identifies the sentence that caused the failure."

Separate essential skills from skills you can teach. If your team can teach a labeling tool, don't reject a candidate solely for using another tool. If the role requires technical judgment without supervision, include a task that tests that judgment.

Define the first months of work

Use milestones to describe increasing responsibility. The schedule below is a planning example; adjust it to the role and onboarding process.

Fill in volume and quality targets after reviewing a representative batch. Record task difficulty and the amount of reviewer help. A target for short text labels should not carry over unchanged to complex technical answers.

Give the manager responsibilities too. Name who supplies examples, answers questions, and resolves disputed scores. A hire cannot meet a clear standard if the team keeps changing it without notice.

Set a scoring rule before interviews

This example suits a role that reviews AI responses. Change the weights before the search if your role has different demands.

Use a four-point scale. A score of one means the candidate cannot complete the task. Two means substantial correction is needed. Three means the work meets the agreed standard. Four means the candidate meets that standard and explains relevant edge cases.

Record an example beside every score. "Good judgment" is too vague. "Flagged conflicting instructions and paused the affected item" gives the hiring manager something to assess.

For an initial decision rule, require at least three on each essential criterion. Use the weighted score to compare candidates who pass those requirements. A high total should not erase a failure on data handling or another mandatory condition.

Calculate the weighted score by multiplying each rating by its weight, then adding the results. Keep the result on the same four-point scale. Record "Not assessed" when a task did not test a criterion, and leave the decision open until that evidence is collected.

Fill in a role-specific example

For a model-response reviewer, the role purpose could be: "Review draft support answers against the approved product policy before release." The main output is a batch of decisions with a reason attached to each rejected or disputed answer.

The work boundary could say: "The reviewer may accept answers that meet the policy and flag answers that need correction. The product owner approves policy changes." This separates reviewing content from deciding what the product should promise.

An essential skill could be finding an unsupported claim in a fluent answer. A teachable skill could be using your internal review queue. Test the first skill directly and include time to learn the queue in onboarding.

Specify what a useful handoff contains. In this example, it names the affected item, quotes the relevant policy, and identifies the decision needed. "Needs review" would not meet the standard because the next person must repeat the investigation.

Keep this example narrow. If the same hire must write code, add a separate coding task with its own acceptance rules. Don't infer engineering skill from a strong response-review score.

What does the data say about defining the role?

Match pay research to the actual work. BLS reports a median annual wage of $135,980 for US software developers in May 2025. It reports $104,300 for software quality assurance analysts and testers over the same period. These are separate occupational measures. BLS Occupational Outlook Handbook

Neither figure is a quote for an AI training hire. Use them as context only when the duties fit those occupations. For broader wage research, start with the BLS Occupational Employment and Wage Statistics program, then select the relevant occupation.

A role that builds evaluation software may need an engineering assessment. A role that applies a written label policy may need a carefully reviewed sample instead. Calling both roles "AI trainer" does not make their budgets or hiring tests comparable.

Keep a compensation worksheet beside the scorecard. Record the job scope, location, seniority, and costs included in each quote. Compare like-for-like totals before calculating savings. A salary figure and a monthly service invoice cover different items.

Build evidence from a shared work sample

Give candidates the same instructions, approved tools, and time allowance. Use synthetic or properly cleared examples. Explain whether AI assistance is allowed and what the candidate must disclose.

For a model-response reviewer, the sample could include straightforward answers, a disputed case, and an answer that appears plausible but contains an error. Ask candidates to record their decision and explain it.

Have your own reviewers complete the sample first. Resolve disagreements in the answer key before using it to judge candidates. When more than one answer is defensible, describe the reasoning you will accept.

Track these fields for each attempt:

• Candidate identifier and scorecard version.
• Decisions accepted against the agreed rubric.
• Errors by type, including their effect on the task.
• Cases escalated and the reason for each escalation.
• Reviewer help required.
• Time spent, with pauses or access problems noted.

Define the denominator for any percentage. If you report accuracy, say which items were scored and how disputed items were handled. Keep serious errors visible even when the overall score looks strong.

Review speed after quality. A candidate who finishes quickly but creates extensive rework may not meet the role's needs. Conversely, someone who pauses to clarify a flawed instruction may be showing the judgment you want.

Make errors easy to compare

Create a short error log alongside the score. For each error, record the candidate's decision, the expected action, and the rule that supports it. Distinguish a missed instruction from a reasonable answer to an unclear question.

For example, a candidate might approve an answer that promises a refund outside the stated policy. The evaluator should identify the promise and the policy conflict. Writing "weak attention to detail" would hide the actual failure.

If several candidates misunderstand the same instruction, review the task before drawing conclusions about their skills. Correct a flawed prompt and reassess affected candidates consistently. Keep the earlier version so evaluators can explain why a score changed.

Test how the candidate uses feedback

After the first pass, give the candidate a correction and ask for a revised answer. Assess whether the revision follows the rule and whether the explanation makes sense.

This is especially useful for entry-level roles. It lets you distinguish a teachable tool gap from repeated failure to apply instructions. Use the same feedback stage for comparable candidates.

Course certificates can guide follow-up questions. They should not replace this evidence. Ask candidates to demonstrate the skills the course claims to teach on a task relevant to your team.

How do hiring models change a nearshore role scorecard?

Use the same outcome standards across hiring models. Change the ownership fields to show who selects the person, supervises work, and resolves quality problems.

The table describes arrangements to confirm during procurement. Put the agreed responsibilities in writing.

If you need an individual working inside your team, evaluate that person's ability to perform the job. If you are buying a completed dataset, evaluate the delivery process and acceptance rules as well.

For outsourced work, ask how the provider handles changes to instructions and disagreements about quality. For a placement, ask who coaches the hire after they start. Those are different responsibilities even when both proposals use similar language.

Searches for the best AI training companies or AI recruitment companies can mix these models together. Narrow the list by the result you need before comparing proposals.

Define nearshore and Latin American working hours

Write the actual meeting window into the role scorecard. "US overlap" leaves room for different expectations about daily schedules and response times.

Bogotá stays on UTC-5 without daylight saving time. Its relationship to US Eastern time changes when the US clock changes. Record the candidate's local schedule and the team's named time zone rather than assuming the offset stays fixed. IANA Time Zone Database

Also state which work can happen outside shared hours. A reviewer may complete a batch independently but need access to a domain expert for disputed cases. Specify how those cases wait, who answers them, and whether they block the next task.

Use a written handoff exercise to test this arrangement. Ask the candidate to summarize completed work, unresolved items, and the decision needed from the next person.

For AI companies hiring remote workers, the scorecard should also name the person who can unblock access problems. Confirm how the hire reports an outage and where work pauses when an approved system is inaccessible. Avoid making response-time promises that depend on accounts the hire does not yet have.

What should you ask a provider before accepting candidates?

Send the completed scorecard with your request. Ask the provider to return evidence against each essential requirement. A profile summary alone should not close an assessment item.

Ask how screening maps to your work

Request a description of the task used to assess candidates. Ask who grades it, what the passing standard is, and how assessors handle disputed answers.

If the provider uses a general AI assessment, identify which parts match your role. Keep your own test for responsibilities that need direct proof. A candidate who can use a writing assistant has not yet demonstrated that they can review technical model answers.

For each finalist, request:

• The evidence supporting each claimed essential skill.
• A clear account of the candidate's contribution to past work.
• Any unresolved skill gaps.
• Confirmed availability for the required schedule.
• The next assessment needed before a decision.

Avoid requesting confidential work from previous employers. A synthetic task or an explanation of a past decision can provide useful evidence without exposing client data.

Ask who owns quality after the start date

Name the person who approves the first batch and the person who resolves disagreements. Establish how the hire receives corrections and where updated instructions live.

Set a review cadence before onboarding. Keep the first workload small enough for the manager to inspect. Increase scope when the evidence supports it.

If quality drops, record the failure before choosing a remedy. Check whether the task changed, the instructions conflict, or the hire needs coaching. Share the relevant evidence with the provider when requesting support.

Carry the hiring rubric into the first review. Compare the work with the same criteria used during selection, then record any new responsibilities separately. This helps the manager distinguish a hiring mistake from a job that changed after the offer.

Ask the hire to explain recurring disagreements. They may reveal an unclear policy or an example that contradicts the written rule. Assign an owner to resolve each issue and record when the revised instruction takes effect.

Ask what the quote and contract cover

Request a written quote tied to the completed role scope. List what the total includes and ask how changes in seniority, duties, or location affect it.

Confirm the equipment and software responsibilities. Ask who approves data access, closes accounts at departure, and retains assessment records. Keep those duties with named owners.

For any replacement promise, ask what starts a new search and what happens to unfinished work. A new candidate search still leaves your manager with a handover and access review to complete.

Run the decision meeting against the evidence

Have evaluators score independently before discussing candidates. Review disagreements by returning to the work sample.

Separate an observed weakness from an untested assumption. If evidence is incomplete, assign a focused follow-up task. Don't turn uncertainty into a favorable score because the interview felt easy.

Use a decision record with fields for the essential criteria, supporting evidence, unresolved questions, and final approval. Keep interview impressions separate from demonstrated skills. A candidate's confidence should not change the score for an answer that fails the written rule.

Close the meeting with a written decision: hire, decline, or collect specific evidence. Record who approves the decision and which conditions must be met before the start date.

Where does Opus fit in this scorecard?

This site is an Opus-owned reference; recommendations are editorial.

For a proposed Opus placement, use the scorecard to define the work your team will manage. Opus states: "The client chooses the finalist and manages day-to-day work." Keep the final assessment and the manager's review duties explicit.

Talk to Opus about a role.

Opus states: "Opus charges one flat monthly rate per hire. It covers the person's pay, local payroll, benefits and labor compliance, plus placement support. There is no separate recruiting fee. The rate depends on role, seniority and location."

Published dollar rate: Not published. Ask for a written quote for the completed role scorecard.

Opus also states: "Every placement carries a lifetime replacement guarantee with no cap and no time limit." Ask how a replacement request would work for your role and who would manage the handover.

Use the same evidence requirements for an Opus finalist as for any other candidate. Request a work sample tied to the job, confirm the required hours, and record any skills that need further assessment.

Before committing, give the hiring manager the completed rubric, candidate evidence, and written quote. Resolve any difference between the work you assessed and the work you expect after hiring. Keep that agreed version for the first performance review.

Talk to Opus about a role.

<!-- File not saved because the workspace is read-only. Example scoring weights and thresholds require calibration to the role. -->