Choice-only RL Models¶

Audience:

• Users modeling choices without response-time likelihoods.
• HSSM users preparing response-only RLSSM data from ssms simulations.

Prerequisites:

• Basic familiarity with Rescorla-Wagner learning and softmax choice rules.
• ssm-simulators installed with compiled simulator extensions.

By the end you will be able to:

• Simulate choice-only Rescorla-Wagner softmax models.
• Explain the beta inverse-temperature parameter.
• Validate response-only data for HSSM handoff.
• Use ssms PPC for choice-only posterior predictive simulation.

Outline¶

1. Load the built-in choice-only presets.
2. Simulate response-only RL data.
3. Inspect the Q-value learning rule.
4. Validate and prepare data for HSSM.
5. Run choice-only PPC and extend to more actions.

In [1]:

from __future__ import annotations

import pandas as pd

import ssms.rl as rl

pd.set_option("display.max_columns", 20)
pd.set_option("display.width", 120)

SEED = 91

from __future__ import annotations import pandas as pd import ssms.rl as rl pd.set_option("display.max_columns", 20) pd.set_option("display.width", 120) SEED = 91

1. What makes a choice-only RL model different?¶

Classic RLSSMs often produce both RT and choice. Choice-only RL models keep the trial-wise learning rule but use a categorical softmax decision process instead of an RT likelihood.

For the built-in inverse-temperature softmax models, the learner emits pre-update Q-values (q0, q1, ...), and the decision process samples choices from softmax(beta * q_values).

In [2]:

rl.preset.list()

rl.preset.list()

Out[2]:

['2AB_RW_Angle', '2AB_RW_InvTempSoftmax', '3AB_RW_InvTempSoftmax']

In [3]:

config = rl.preset.get("2AB_RW_InvTempSoftmax")
print(rl.preset.info("2AB_RW_InvTempSoftmax"))

config = rl.preset.get("2AB_RW_InvTempSoftmax") print(rl.preset.info("2AB_RW_InvTempSoftmax"))

Preset: 2AB_RW_InvTempSoftmax
Description: Two-armed bandit with a Rescorla-Wagner delta-rule learner and a choice-only inverse-temperature softmax decision process.
Task: two-armed Bernoulli bandit
Learning process: RescorlaWagnerSoftmax
Decision process: inv_temp_softmax_2
Required parameters: rl_alpha, beta
Default parameters: rl_alpha=0.2, beta=1
Response labels: (0, 1)
Response to choice: {0: 0, 1: 1}
Context fields: ['feedback']
Learning backend: jax
Gradient support: available
HSSM participant contract: yes

2. Simulate a two-armed choice-only model¶

The required parameters are just the learning rate rl_alpha and inverse temperature beta. Larger beta makes choices more deterministic for the currently higher-valued option.

In [4]:

theta = {"rl_alpha": 0.20, "beta": 2.0}

sim = rl.Simulator(config)
data = sim.simulate(
    theta=theta,
    n_trials=120,
    n_participants=5,
    random_state=SEED,
)

print(data.shape)
data.head()

theta = {"rl_alpha": 0.20, "beta": 2.0} sim = rl.Simulator(config) data = sim.simulate( theta=theta, n_trials=120, n_participants=5, random_state=SEED, ) print(data.shape) data.head()

(600, 5)

Out[4]:

	participant_id	trial_id	rt	response	feedback
0	0	0	-1.0	0	1.0
1	0	1	-1.0	0	1.0
2	0	2	-1.0	0	1.0
3	0	3	-1.0	1	1.0
4	0	4	-1.0	0	0.0

In [5]:

print("Columns:", data.columns.tolist())
print("Unique rt placeholders:", sorted(data["rt"].unique()))

choice_summary = (
    data.groupby("participant_id")
    .agg(
        n_trials=("trial_id", "count"),
        p_choice_1=("response", lambda x: (x == 1).mean()),
        mean_feedback=("feedback", "mean"),
    )
    .round(3)
)
choice_summary

print("Columns:", data.columns.tolist()) print("Unique rt placeholders:", sorted(data["rt"].unique())) choice_summary = ( data.groupby("participant_id") .agg( n_trials=("trial_id", "count"), p_choice_1=("response", lambda x: (x == 1).mean()), mean_feedback=("feedback", "mean"), ) .round(3) ) choice_summary

Columns: ['participant_id', 'trial_id', 'rt', 'response', 'feedback']
Unique rt placeholders: [np.float64(-1.0)]

Out[5]:

	n_trials	p_choice_1	mean_feedback
participant_id
0	120	0.392	0.558
1	120	0.342	0.550
2	120	0.292	0.600
3	120	0.283	0.592
4	120	0.367	0.600

The rt column is present only because the low-level simulator interface has historically returned RT arrays. For choice-only models, every value is the non-omission placeholder -1.0; it is not an observed response time.

3. Inspect the learning rule¶

RescorlaWagnerSoftmax emits Q-values before each update. After the observed response and feedback, the chosen action is updated by the Rescorla-Wagner delta rule.

In [6]:

learner = rl.learning.RescorlaWagnerSoftmax(n_actions=2, initial_q=0.5)
state = learner.init_state()
rows = []

first_subject = data[data["participant_id"] == 0].head(10)
for _, trial in first_subject.iterrows():
    q_before = learner.compute_python(state, {"rl_alpha": theta["rl_alpha"]}, {})
    rows.append(
        {
            "trial_id": int(trial["trial_id"]),
            **q_before,
            "response": int(trial["response"]),
            "feedback": float(trial["feedback"]),
        }
    )
    state = learner.update_python(
        state,
        {"rl_alpha": theta["rl_alpha"]},
        {"choice": int(trial["response"]), "feedback": float(trial["feedback"])},
    )

pd.DataFrame(rows)

learner = rl.learning.RescorlaWagnerSoftmax(n_actions=2, initial_q=0.5) state = learner.init_state() rows = [] first_subject = data[data["participant_id"] == 0].head(10) for _, trial in first_subject.iterrows(): q_before = learner.compute_python(state, {"rl_alpha": theta["rl_alpha"]}, {}) rows.append( { "trial_id": int(trial["trial_id"]), **q_before, "response": int(trial["response"]), "feedback": float(trial["feedback"]), } ) state = learner.update_python( state, {"rl_alpha": theta["rl_alpha"]}, {"choice": int(trial["response"]), "feedback": float(trial["feedback"])}, ) pd.DataFrame(rows)

Out[6]:

	trial_id	q0	q1	response	feedback
0	0	0.5000	0.50000	0	1.0
1	1	0.6000	0.50000	0	1.0
2	2	0.6800	0.50000	0	1.0
3	3	0.7440	0.50000	1	1.0
4	4	0.7440	0.60000	0	0.0
5	5	0.5952	0.60000	1	0.0
6	6	0.5952	0.48000	1	1.0
7	7	0.5952	0.58400	1	0.0
8	8	0.5952	0.46720	1	1.0
9	9	0.5952	0.57376	0	0.0

4. Validate response-only data¶

Drop the placeholder rt column before HSSM handoff or choice-only PPC. The model config declares response=["response"], so validation expects participant IDs, responses, and context fields such as feedback.

In [7]:

response_only = data.drop(columns=["rt"])
report = config.validate_data(response_only)
report.print()
report.raise_for_errors()
response_only.head()

response_only = data.drop(columns=["rt"]) report = config.validate_data(response_only) report.print() report.raise_for_errors() response_only.head()

RLSSM data validation report:
  [WARNING] extra_columns: data has extra columns not required by this model: ['trial_id'].
  panel shape: participants=5, trials_per_participant=120

Out[7]:

	participant_id	trial_id	response	feedback
0	0	0	0	1.0
1	0	1	0	1.0
2	0	2	0	1.0
3	0	3	1	1.0
4	0	4	0	0.0

HSSM handoff uses response-only data:

import hssm

hssm_config = hssm.rl.RLSSMConfig.from_ssms_model("2AB_RW_InvTempSoftmax")
model = hssm.RLSSM(data=response_only, model_config=hssm_config)

The ssms learning rule supplies the trial-wise Q-value trajectory used by the HSSM RLSSM likelihood.

5. Choice-only posterior predictive simulation¶

For choice-only PPC, observed_data must also be response-only. The simulator replays the observed response/feedback sequence to condition learning, then samples new responses from the softmax decision process.

In [8]:

ppc = sim.simulate(
    theta=theta,
    mode="ppc",
    observed_data=response_only,
    random_state=SEED + 1,
)

print("PPC columns:", ppc.columns.tolist())

pd.DataFrame(
    {
        "observed_response": response_only["response"].head(12),
        "ppc_response": ppc["response"].head(12),
        "feedback_replayed": ppc["feedback"].head(12),
    }
)

ppc = sim.simulate( theta=theta, mode="ppc", observed_data=response_only, random_state=SEED + 1, ) print("PPC columns:", ppc.columns.tolist()) pd.DataFrame( { "observed_response": response_only["response"].head(12), "ppc_response": ppc["response"].head(12), "feedback_replayed": ppc["feedback"].head(12), } )

PPC columns: ['participant_id', 'trial_id', 'response', 'feedback']

Out[8]:

	observed_response	ppc_response	feedback_replayed
0	0	1	1.0
1	0	0	1.0
2	0	1	1.0
3	1	0	1.0
4	0	0	0.0
5	1	0	0.0
6	1	0	1.0
7	1	1	0.0
8	1	1	1.0
9	0	0	0.0
10	0	1	1.0
11	1	0	0.0

6. Three- and four-choice variants¶

ssms ships two- and three-armed choice-only RL presets. The lower-level inv_temp_softmax_4 decision process is also available for custom four-choice configs.

In [9]:

config3 = rl.preset.get("3AB_RW_InvTempSoftmax")
sim3 = rl.Simulator(config3)
data3 = sim3.simulate(
    theta=theta,
    n_trials=90,
    n_participants=3,
    random_state=SEED + 2,
)

data3["response"].value_counts(normalize=True).sort_index().round(3)

config3 = rl.preset.get("3AB_RW_InvTempSoftmax") sim3 = rl.Simulator(config3) data3 = sim3.simulate( theta=theta, n_trials=90, n_participants=3, random_state=SEED + 2, ) data3["response"].value_counts(normalize=True).sort_index().round(3)

Out[9]:

response
0    0.530
1    0.248
2    0.222
Name: proportion, dtype: float64

In [10]:

config4 = rl.ModelConfig(
    model_name="4AB_RW_InvTempSoftmax_Demo",
    description="Four-armed Rescorla-Wagner softmax demonstration model.",
    decision_process="inv_temp_softmax_4",
    learning_process=rl.learning.RescorlaWagnerSoftmax(n_actions=4, initial_q=0.5),
    task_environment=rl.env.Bandit.bernoulli(
        probabilities=[0.55, 0.25, 0.15, 0.05],
        response_labels=[0, 1, 2, 3],
    ),
    response=["response"],
)

sim4 = rl.Simulator(config4)
data4 = sim4.simulate(
    theta=theta,
    n_trials=80,
    n_participants=2,
    random_state=SEED + 3,
)

data4["response"].value_counts(normalize=True).sort_index().round(3)

config4 = rl.ModelConfig( model_name="4AB_RW_InvTempSoftmax_Demo", description="Four-armed Rescorla-Wagner softmax demonstration model.", decision_process="inv_temp_softmax_4", learning_process=rl.learning.RescorlaWagnerSoftmax(n_actions=4, initial_q=0.5), task_environment=rl.env.Bandit.bernoulli( probabilities=[0.55, 0.25, 0.15, 0.05], response_labels=[0, 1, 2, 3], ), response=["response"], ) sim4 = rl.Simulator(config4) data4 = sim4.simulate( theta=theta, n_trials=80, n_participants=2, random_state=SEED + 3, ) data4["response"].value_counts(normalize=True).sort_index().round(3)

Out[10]:

response
0    0.394
1    0.206
2    0.169
3    0.231
Name: proportion, dtype: float64

Exercise¶

Try changing beta while keeping rl_alpha fixed. Lower beta produces more exploratory choices; higher beta makes the model more likely to pick the currently best-valued option.

In [11]:

low_beta = sim.simulate(
    theta={"rl_alpha": 0.20, "beta": 0.5},
    n_trials=120,
    n_participants=5,
    random_state=SEED,
)
high_beta = sim.simulate(
    theta={"rl_alpha": 0.20, "beta": 6.0},
    n_trials=120,
    n_participants=5,
    random_state=SEED,
)

pd.DataFrame(
    {
        "beta_0.5": low_beta["response"].value_counts(normalize=True).sort_index(),
        "beta_6.0": high_beta["response"].value_counts(normalize=True).sort_index(),
    }
).round(3)

low_beta = sim.simulate( theta={"rl_alpha": 0.20, "beta": 0.5}, n_trials=120, n_participants=5, random_state=SEED, ) high_beta = sim.simulate( theta={"rl_alpha": 0.20, "beta": 6.0}, n_trials=120, n_participants=5, random_state=SEED, ) pd.DataFrame( { "beta_0.5": low_beta["response"].value_counts(normalize=True).sort_index(), "beta_6.0": high_beta["response"].value_counts(normalize=True).sort_index(), } ).round(3)

Out[11]:

	beta_0.5	beta_6.0
response
0	0.532	0.897
1	0.468	0.103

Summary¶

• Choice-only RL models keep trial-wise learning but omit the RT likelihood.
• RescorlaWagnerSoftmax emits Q-values, and beta controls softmax sensitivity.
• Generative choice-only output contains rt=-1.0 only as a compatibility placeholder.
• HSSM handoff and ssms choice-only PPC should use response-only data.