Choicerule Models (action-selection rules)
choicerules.RdModels discrete action selection: applies a choce rule/decision rule/action selection rule to select among values.
softmax()fits a soft-maximum = soft approximation to argmax (Sutton & Barto, 2018).epsilon_greedy()fits epsilon-greedy.epsilon()fits probabilistic-epsilon.argmax()maximizes deterministically.luce()selects proportionally (Luce's rule aka Luce's axiom, Luce, 1959).
Arguments
- formula
A formula, the variables in
datato be modeled. For example,y ~ x1 | x2models response y as function of two stimuli with values x1 and x2 (respectively). Lines|separate stimuli.- data
A data frame, the data to be modeled.
- fix
(optional) A list with parameter-value pairs of fixed parameters. If missing all free parameters are estimated. If set to
"start"all parameters are fixed to their start values. Model parameter names aretau(see details - model parameters).list(tau = 3.85)sets parametertauequal to 3.85."start"sets all parameters equal to their initial values (estimates none). Useful for building a first test model.
- options
(optional) A list, list entries change the modeling procedure. For example,
list(lb = c(k=0))changes the lower bound of parameter k to 0, orlist(fit_measure = "mse")changes the goodness of fit measure in parameter estimation to mean-squared error, for all options, see cm_options.- ...
other arguments, ignored.
- discount
A number, how many initial trials to not use during parameter fitting.
Value
Returns a cognitive model object, which is an object of class cm. A model, that has been assigned to m, can be summarized with summary(m) or anova(m). The parameter space can be viewed using pa. rspace(m), constraints can be viewed using constraints(m).
Details
This is how the model predicts and treats observations:
For
formula = y ~ x1andy ~ x1 | x2it predicts the probability to select x1, thusy = 1must mean select x1.For
formula = y ~ x1 | x2 | x3it predicts three columns, the probabilities to select x1, x2, and x3, respectively.
Model Parameters
Most models have no free parameters, except softmax and epsilon greedy which have 1 free parameter each:
In
softmax():tau: the softness of the choices, high values cause more equiprobable choices.In
epsilon()andepsilon_greedy():eps: the error proportion of the choices, high values cause more errors.
Background
epsilon() picks action \(i\) with probability \((1 - \epsilon)*p(i)\) and with \(\epsilon\) it picks randomly from all actions. For \(\epsilon = 0\) it gives \(p(i)\), that is the original probabilistic policy.
epsilon_greedy() picks the best action with probability \(1 - \epsilon\), and with \(\epsilon\) it picks randomly from all actions, including the best.
argmax() picks the highest-valued action with probability 1, and in case of ties it picks equiprobable.
luce() picks action \(i\) with probability \(v(i) / \sum v\).
References
Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd Ed.). MIT Press, Cambridge, MA. (http://incompleteideas.net/book/the-book-2nd.html)
Luce, R. D. (1959). On the possible psychophysical laws. Psychological Review, 66(2), 81-95. doi:10.1037/h0043178
Author
Jana B. Jarecki, jj@janajarecki.com
Examples
# Make some fake data
D <- data.frame(a = c(.3,.8,.5), # value of option A
b = c(.7,.2,.5), # value of option B
y = c(0,1,1)) # respondent's choice (0=A, 1=B)
M <- softmax(y ~ a | b, D, c(tau=1)) # creates soft-max model w tau=1
predict(M) # predict action selection
#> [1] 0.4013123 0.6456563 0.5000000
M$predict() # -- (same) --
#> [1] 0.4013123 0.6456563 0.5000000
summary(M) # summarize
#>
#> Model:
#> with no choice rule
#> Call:
#> y ~ a | b
#>
#> No Free Parameters
#>
#> Fit Measures:
#> MSE: 0.18, LL: -1.6, AIC: 3.3, BIC: 3.3
#>
anova(M) # anova-like table
#> Sum Sq. Table
#> N Par Sum Sq Mean Sq
#> 0 0.53661 0.17887
coef(M) # free parameter (NULL)
#> NULL
M$get_par() # fixed parameter (tau = 1)
#> tau
#> 1
M$npar() # 1 parameter
#> [1] 0
M$MSE() # mean-squared error
#> [1] 0.1788703
logLik(M) # log likelihood
#> 'log Lik.' -1.64365 (df=0)
### Parameter specification and fitting ---------------------------------
softmax(y ~ a | b, D, fix="start") # fix 'tau' to its start value
#> softmax | choice rule: none
#> Call:
#> softmax(formula = y ~ a | b, data = D, fix = "start")
#>
#> Constrained and fixed parameters:
#> tau
#> 2
#>
#> ---
#> Note: No free parameters. View constraints by constraints(.), view parameter space by parspace(.)
softmax(y ~ a | b, D, fix=c(tau=0.2)) # fix 'tau' to 0.2
#> softmax | choice rule: none
#> Call:
#> softmax(formula = y ~ a | b, data = D, fix = c(tau = 0.2))
#>
#> Constrained and fixed parameters:
#> tau
#> 0.2
#>
#> ---
#> Note: No free parameters. View constraints by constraints(.), view parameter space by parspace(.)
softmax(y ~ a | b, D) # fit 'tau' to data y in D
#> Fitting free parameters [tau] by maximizing loglikelihood (binomial pdf) with auto.
#> softmax | choice rule: none
#> Call:
#> softmax(formula = y ~ a | b, data = D)
#>
#> Free parameters: estimates
#> tau
#> 1e-04
#>
#>
#> ---
#> Note: No fixed parameter.
### The different choice rules ------------------------------------------
softmax(y ~ a | b, D, fix=c(tau=0.5)) # fix 'tau' to 0.5
#> softmax | choice rule: none
#> Call:
#> softmax(formula = y ~ a | b, data = D, fix = c(tau = 0.5))
#>
#> Constrained and fixed parameters:
#> tau
#> 0.5
#>
#> ---
#> Note: No free parameters. View constraints by constraints(.), view parameter space by parspace(.)
softmax(y ~ a | b, D) # fit 'tau' to y
#> Fitting free parameters [tau] by maximizing loglikelihood (binomial pdf) with auto.
#> softmax | choice rule: none
#> Call:
#> softmax(formula = y ~ a | b, data = D)
#>
#> Free parameters: estimates
#> tau
#> 1e-04
#>
#>
#> ---
#> Note: No fixed parameter.
epsilon_greedy(y~a | b, D, c(eps=0.1)) # fix 'eps' to 10 %
#> epsilongreedy | choice rule: none
#> Call:
#> epsilon_greedy(formula = y ~ a | b, data = D, fix = c(eps = 0.1))
#>
#> Constrained and fixed parameters:
#> eps
#> 0.1
#>
#> ---
#> Note: No free parameters. View constraints by constraints(.), view parameter space by parspace(.)
epsilon_greedy(y~a | b, D ) # fit 'eps' to y
#> Fitting free parameters [eps] by maximizing loglikelihood (binomial pdf) with auto.
#> epsilongreedy | choice rule: none
#> Call:
#> epsilon_greedy(formula = y ~ a | b, data = D)
#>
#> Free parameters: estimates
#> eps
#> 0
#>
#>
#> ---
#> Note: No fixed parameter.
epsilon(y ~ a | b, D, c(eps=0.1)) # fix 'eps' to 0.1
#> epsilon | choice rule: none
#> Call:
#> epsilon(formula = y ~ a | b, data = D, fix = c(eps = 0.1))
#>
#> Constrained and fixed parameters:
#> eps
#> 0.1
#>
#> ---
#> Note: No free parameters. View constraints by constraints(.), view parameter space by parspace(.)
epsilon(y ~ a | b, D) # fit 'eps' to y
#> Fitting free parameters [eps] by maximizing loglikelihood (binomial pdf) with auto.
#> epsilon | choice rule: none
#> Call:
#> epsilon(formula = y ~ a | b, data = D)
#>
#> Free parameters: estimates
#> eps
#> 0
#>
#>
#> ---
#> Note: No fixed parameter.
luce(y ~ a | b, D) # Luce's choice rule, 0 parameter
#> luce | choice rule: none
#> Call:
#> luce(formula = y ~ a | b, data = D)
#>
#>
#> ---
#> Note: No free parameters. No fixed parameter.
argmax(y ~ a | b, D) # Argmax choice rule, 0 parameter
#> argmax | choice rule: none
#> Call:
#> argmax(formula = y ~ a | b, data = D)
#>
#>
#> ---
#> Note: No free parameters. No fixed parameter.