Business Data Analyst (AI Workflow Optimization Specialist)

About the Role

You will own the quality and accuracy of AI-assisted fraud and risk review workflows. Working at the intersection of LLM prompt engineering and operations, you will analyze where the AI agent disagrees with human analysts, diagnose root causes, and iteratively improve the prompts and decision logic that power the agent.

What You Will Do

• Evaluate AI output quality — read model decisions and reasoning, identify where the AI diverges from expected outcomes, and prioritize gaps to fix.
• Improve the AI instructions — edit structured prompt and rubric files to sharpen decision logic, close coverage gaps, and prevent regressions.
• Run controlled experiments — propose a change, validate it against a representative test set, and ship only when results meet the quality bar.
• Maintain ground-truth test coverage — keep the verified test suite current as the SOP evolves; add new cases when new edge cases surface.
• Interpret operational metrics — track agreement trends over time, spot patterns in failures, and use data to guide where to focus next.
• Close the feedback loop — gather input from human reviewers, translate observations into concrete improvements, and confirm fixes hold.

What You Will Need

Required

• Strong analytical and written communication skills — you will spend most of your time reading AI reasoning, writing precise instructions, and explaining trade-offs.
• Comfort working with LLM outputs — reading model reasoning chains, spotting hallucinations, identifying ambiguous instructions.
• Ability to effectively work with AI-based IDEs (such as Cursor or Claude Code) to read, edit, and validate configuration files.
• Systematic, hypothesis-driven mindset — you test one thing at a time and measure before shipping.
• High attention to detail in regulatory or risk-adjacent domains (fraud, compliance, underwriting, identity).

Nice to Have

• Background in fraud operations, compliance review, or financial services risk.
• Prior experience with prompt engineering or iterative AI improvement workflows.
• Familiarity with A/B experimentation concepts (champion vs. challenger, statistical significance).
• Ability to read basic Python to understand scripts you will interact with.