Week 2 · Directing and Supervising AI Agents

Kopi Kembali’s marketing lead wants to know what actually works on social media. The point of this lab is not the answer. It is the instruction. You will run the same analysis twice, once with a vague prompt and once with a structured one, and compare what comes back.

At a glance

Time	~60 minutes, in class
Work with	A partner
You need	An agent (Manus, or Claude Code as your instructor confirms) + the dataset
You hand in	Your Two-Step Agent Task record: two prompts, two outputs, and your supervision notes
Graded on	Unmarked LO1 evidence — did you follow the four prompt patterns, and can you reflect honestly on what worked

The scenario

You have twelve weeks of the chain’s posts across six platforms. Treat every column the way you would treat a colleague’s export: worth profiling before it is worth trusting.

Data dictionary

Column	Meaning
post_id	Unique post id
platform	Instagram, TikTok, Facebook, LinkedIn, Twitter or YouTube
post_date, post_time, day_of_week	When the post went out
content_type	video, carousel, photo or text
likes, comments, shares	Engagement actions
views	Views or impressions for the post
follower_count	Platform followers at the time
engagement_rate_pct	A pre-calculated rate. The formula is not documented anywhere
hashtag_count	Hashtags used
sentiment	Sentiment tag applied by an earlier tool

Undocumented formula

engagement_rate_pct is pre-calculated, but nobody has written down how it is computed. Treat it as a claim to verify, not a fact — in both runs below, notice where the agent (or you) leans on it without asking what it means.

Get your data

• Download kk-social-posts.csv

Step 1 — The naive run · 10 min

Give the agent the file and exactly this prompt. Do not steer it, and save the full response.

Analyse this social media data and tell me what works.

Step 2 — The structured run · 20 min

Start a fresh session so nothing leaks from Step 1. First, as a pair, pick one business question — specific enough that the answer could guide a real decision:

• Which platform has the highest median engagement rate?
• What time of day gets the most views across all platforms?
• Which content type drives the most shares relative to views?
• How does sentiment relate to engagement on Instagram versus TikTok?

Write it down before you design your prompt. Then run these two prompts in order, pausing after each to read the response.

Prompt 1 — profile first (do not analyse yet):

Profile this dataset. List every column, its data type, the share of missing
values, and anything that looks inconsistent or pre-calculated. Do not
analyse yet, just describe.

Prompt 2 — answer your business question. Fill in the blank with the question you chose above:

Now answer this question: [your business question]. Show the calculation or
grouping you used so I can verify it.

🔗 On your agent platform

A worked example of the same two-step pattern, if you want to see it applied to this scenario before writing your own: “You are a marketing analyst for a Singapore kopi chain. Profile this dataset first: list every column with its type, row count, missing values, and any inconsistencies, then stop.” — followed by “Using only rows with complete views data, compare mean and median engagement actions by content type, and state which platform drives any difference. Present a table. Flag any figure a sceptical reader should check.”

Step 3 — The comparison · 20 min

Put the two outputs side by side and note:

• What did the naive run assert without evidence?
• What did the structured run surface that the naive run missed?
• Where did either run rely on engagement_rate_pct without asking what its formula is?

Then run the structured output through this check before you trust it:

• Hallucination audit — did the agent invent any metrics, categories, or numbers not in the data?
• Data lineage — can you point to exactly which column or calculation produced the final number?
• Reasonableness check — do the numbers make sense against what you saw in the profile?
• Escalation flag — did the agent flag missing data, inconsistencies, or small sample sizes? If not, and you noticed problems in the profile, that is a reason to ask for clarification.

Step 4 — The record · 10 min

Write your Two-Step Agent Task record. Both prompts verbatim, not paraphrased.

TWO-STEP AGENT TASK RECORD
Team: ______   Date: ______
Business question: ______________________________________________

PROMPT 1 (profile) — verbatim:


PROMPT 2 (business question) — verbatim:


NAIVE RUN vs STRUCTURED RUN
  What the naive run asserted without evidence:
  What the structured run surfaced that the naive run missed:
  Where either run leaned on engagement_rate_pct unverified:

REFLECTION
  What did the agent do right? (one sentence)
  What would you change if you ran this again? Be specific — "the prompt was
  too vague about platform names" beats "it did not work."
  Did anything in the profile change how you interpret the answer?

SUMMARY (one sentence): what you will always include in an analytics prompt
from now on.

🔗 Optional — on Claude Code

For terminal-comfortable students: write one plain-English instruction for a data task (e.g. “analyse this CSV and tell me the top five values in each numeric column”), point it at a small public CSV, and watch Claude Code write and run the Python. Read the code — you do not need to understand every line, just note whether it does what you asked. If the output is wrong, tell the agent what went wrong and let it fix the code. This is optional and does not replace Steps 1–4.

Verify & submit

• Both prompts recorded exactly as sent, not paraphrased
• The profile’s row count matched the file when you opened it yourself
• You can name at least one claim in the naive output you would not repeat to the marketing lead
• Your record says what the structured prompt controlled, not just that it was better
• Self-critique is filled out completely

Add to your individual portfolio, clearly labelled:

1. Prompt 1 (profile) and Prompt 2 (business question)
2. Output 1 (profile result) and Output 2 (answer result) — screenshot or text
3. Self-critique — your reflection section, filled out completely

Capstone connection

You will choose your capstone dataset in Unit 03. Before you use an agent on it, work through the same governance questions you are practising here:

• Privacy — Does this data contain personal information? Do you have permission to share it with an agent service?
• Confidentiality — Is this data confidential to your organisation? Does it need to stay off hosted services?
• Hallucination and error risk — What could the agent get wrong with this data, and how serious would that be?
• Disclosure — Will you tell people you used an agent? Have you planned how to disclose that?
• Data residency — If you use a hosted service, where will the data be processed? Is that acceptable?

Unit 02 app · lesson plan