Uhm, Helen. Are you recording?
Yes, I just started.
Excellent thank you.
Alright so I am extremely excited
to welcome Ryan Smith to the next
seminar this month where I met
Ryan was about couple years ago.
Now I think when I visited a library
and Ryan from that time on to be
not only extremely well read and
knowledgeable about everything
computational but also really fun to
work with and then tried to call again.
Ryan attended college at Arizona
State and then completed three
graduate degrees, just three.
Two masters in neuroscience and philosophy
before completing a PhD in psychology,
during which you worked with Richard
Lane and he focused on providing
cortex in motion and pathology
and major depressive disorder.
And then you did a postdoc at also
in Arizona with Fat Kilgore where he
focused on emotional working memory
and after that he took a position as
an associate investigator with golf,
wrist and UCL.
And there he focused on designing
computational models of brain
function for use in understanding
emotion and psychopathology.
And since that time Ryan has
continued to be prolific contributions
ranging from the highly theoretical
to the extremely practical.
And that includes the tutorial and active
inference model will be focusing on today.
I'm always inspired by as well.
I find it really,
really fun to actually work with them,
and I'm really excited to
welcome him to maps today.
So Ryan welcome.
OK well yeah. I mean,
I really want to thank Alan Helen.
First of all, just for inviting
me and having the opportunity.
You know, I really hope at
least some of this comes off
comprehensible and useful to you all.
And since it's recorded, obviously
you're gonna watch it multiple times.
So maybe eventually it will be.
But yes. So just kind of jumping
in the topic here it is.
Active inference,
which is a particular sort of
computational modeling approach that
we've tried to make more accessible with.
Tutorial material we've put out.
You know,
I should note that I'm calling this an
introduction to active inference here.
That's probably much more.
Appropriately called Crash
course on active inference,
so we are trying to kind of cover a lot
here over these two and a half hours.
Starting from,
you know,
trying to assume you know fairly little
in terms of previous mathematical
background by the audience,
and get kind of all the way through to,
you know,
giving some tools for actually using this.
This sort of modeling approach in practice,
so I'll start by giving a kind
of conceptual introduction to
active inference for people
who aren't familiar as well as.
Kind of interleaved in there some
bit of a primer on the mathematics of
probability and Bayesian inference,
and then I'll go into the
active inference formalism.
We're all kind of heavy math components are,
which includes a number
of a number of things.
So graphical models message passing schemes.
The neural process theory
associated with active inference,
the notions of or the constructs of
variational and expected free energy.
Then the player really central
role in active inference and
decision making and perception.
As well as how those contribute
to policy selection or selecting
sequences of actions or plans and
and then and then active learning.
So the way that we model learning
and active inference,
which ends up playing very,
you know,
major role and you're trying to model
most kind of like reinforcement learning,
your decision,
making tasks that involve learning and say,
reward probabilities and things like that.
And then once you've covered all of that,
then I'm in the second part.
I'll move on to empirical applications
which will involve examples.
And also providing sort of
tools for making behavioral and
neural predictions using models,
how to fit models to behavioral data?
And if there's time and I'm
really hoping there is,
I'll be able to walk you through
some of the example MATLAB code
that we've that we've provided
that will give you kind of should
give you the kind of skeleton of
what you need to do to apply this
sort of thing to your own studies.
So, OK,
so just a conceptual introduction
for perceptual active inference.
So what is active inference?
So this can be a little confusing
in the literature actually,
because the term was initially used
in older literature to refer to a
theory of predictive motor control,
which was more or less just an extension
of predictive processing models of
perception like predictive coding.
And there it was basically just a I mean a
very interesting you know model of how you
can use predictions to act as motor commands.
But more recently, since around 2015,
active inference is now more often used
to a different theory that's related.
But it's a theory of predictive
decision making as a motor control,
and that's what we're covering
in this lecture.
Both of these theories are grounded in
Bayesian inference or in probability theory.
As we talk about,
one is just deciding what to do,
which is what we're talking about,
whereas the other is about
controlling the body to enact the
decision after it's been made.
But the basic idea with active inference,
and I'm going to give a lot of
examples of this, is that agents
don't simply passively observe.
They're not passive observers,
even in the probabilistic inference sense,
like in predictive coding.
If you're familiar with that,
and instead agents also actively infer
the probability of future observations,
and given the different options
that they might, they might choose.
So, for instance,
they might predict if I choose to go inside,
then I'll feel warm and it comes.
And then you combine that with
probabilistic predictions about preferences.
So I want to feel warm,
therefore I'll go inside.
So that would be your preference and then.
And this is a really central
aspect of active inference.
Is the agents also select actions to
gather information that will help
them be more confident about how to
achieve preferences in the future.
So for instance,
if you're going to bumbling out
in the woods in the dark,
and you're trying to find your
house because that's where you want
to go to feel warm, you might say,
OK,
I'm going to turn on my flashlight
because I predict that's going to
help me figure out how to get inside.
So active inference models actually
do that kind of thing pretty well,
and and so this is kind of a kind
of figure you might see in papers.
It's really scary, right?
It shows all these equations with
all these variables that it's not
clear what they mean and then kind of
graphical model depiction over here.
And again,
this can be pretty daunting to most.
People when they try to get into
this area to begin with.
Very understandably so and and so eventually,
though,
what I'm hoping is that by the end of this,
this will look a little bit more
comprehensible and and they'll
be able to have enough of a grasp
on it to start using it.
So and before I go into more detail,
I really want to motivate this,
you know,
by by talking about what active
inference models are actually good for.
So the first thing that's useful about
them is that they incorporate perception,
learning,
and decision making within within
a single model.
And so it allows the framework
to a plot to be applied to a
fairly wide range of problems,
including, you know, simple,
sort of like classic perception tasks,
as well as reinforcement learning
tasks and planning tasks.
And another useful thing is is that
the the equations that I just showed
you the equation is used in active
inference or are fairly generic.
So in other words you don't have to change.
The equations when you're applying them
to different generative model structures.
So if you build a model of a task,
then different task models will still
use those same generic equations,
so the the the main kind of task when
you're using these models in studies is
just to figure out what the right model
structure is to capture and generate
the behavior that you're interested in
within a particular behavioral task.
Another useful thing about active inference
models is they're they're motivated,
kind of right from the start.
By being biologically plausible.
In other words, there's there's clear ways
in which neural networks could implement
all the linked equations in the model,
and as a result,
it has an accompanying neural process theory,
which this is one way of depicting
a possible implementation.
And again,
I'll I won't go through this
in detail right now,
but the basic idea is just that you
can set up a set of artificial neurons.
These little kind of bolner on things
here with particular patterns of synaptic
connections that just do addition
and subtraction and multiplication.
Uhm,
and they're able to implement and carry out
each of those linked equations I showed,
and so it's and so how loud you too.
It gives you the opportunity to
generate and test really precise
hypotheses and predictions about that.
It could be used in neuroimaging studies,
so things like generating simulated
ERP's or simulating response
patterns like time series patterns
that could be used as regressors
in an fMRI experiment for example.
And the last kind of major thing that's
pretty unique about active inference,
or that's useful about it,
is it provides a unique approach for
modeling. Explore, exploit, tradeoffs.
System kind of mentioned a little
bit earlier,
but it's a arbitrates whether a
person decides to seek out reward
directly just to go straight for
their goal or try to decide what
to do to their goal or to kind of
first seek out more information.
So that you can have a better sense
of what it is you should do to get
to your goal most effectively,
and so it can be a useful framework
specifically for modeling the kind
of behavioral tasks that involve
this kind of information.
Seeking and planning aspects to them.
And as a result,
you know we and other people have
started to use these models in
empirical studies that that have
these sorts of aspects to them,
and I won't talk about them now
more than just showing they exist,
but here's a few studies empirical.
Participant on on information
speaking there's a couple studies
that we have done on planning in
the context of approach avoidance,
conflict and then a couple theory,
a couple papers. So one by us and
one by others fell short in back.
In particular, testing Neuro
scientific predictions in fMRI EG.
So, so going forward,
I just want to acknowledge right
from the start that most of
this talk will draw from the
recent tutorial that we put out,
and so I really need to
acknowledge my Co authors on that.
So one being Karl Friston,
who is the was the main original person
who came up with and sort of articulated
active in for the active inference
framework and then Christopher White,
who's just an incredibly talented
grad student that that I worked
with a fair amount now.
OK, so first let's start
with perceptual inference.
Because to understand active inference,
we need to understand something about
probabilistic perceptual inference
and and the idea here with perceptual
inference is we're trying to figure out,
given some sensory input,
what interpretation is most likely.
The reason that's really important is
because sensory input is just always
consistent with more than one interpretation,
and so the brain needs to bring something
else to the table to figure out which
of all the possible interpretations is.
Is the best one and what
it brings to the table.
Is this idea of part is our prior belief,
so it has some model of the world that
says these sorts of interpretations
are just more likely, you know,
apriori than these other interpretations.
And just just as kind of a
simple example of this,
you can see here in this image on the
top that the image here on the retina.
This kind of upside down leaf image.
That image will be identical given,
say, a small leaf that's close to you.
And, uh, you know gigantic leaf.
That's a little farther away.
So if it's the same image on the retina,
then the brain essentially is
the same information.
So to arbitrate which one it is,
you need to do some kind of
inference process that brings
prior expectations to the table.
And so basically we want to know
what the probability is that it's the
small leaf versus a largely after
receiving that retinal image and in
sort of the probabilistic notation
for this will we be saying is that
we wanna know the probability of
each leaf side is given the retinal
image where this vertical bar here
in notes conditionality.
So this is the conditional probability
of the leaf size given the retinal image
that we know what the retinal image is.
And we can generalize this to any sort of
inference process where we want to know
the probability of the state of the world,
given that we just got some new observation.
And often in Bayesian inference this is
referred to as a posterior probability.
So it's a probability your
probabilistic belief about the state
of after getting some observation.
And to calculate this posterior belief
we need some kind of model of the world
that includes two different elements.
First, we want to know what sorts
of images do small and large the
leaves tend to generate,
so that would be the inverse
of the posterior.
Here should be the probability of different
retinal images given different leaf sizes,
or generically the probability of
an observation given some state
of the world and.
It's often called the likelihood mapping.
In Bayesian inference.
The other thing I model needs to have
is a prior belief about how common large
versus small leaves are in general,
so that would just be the probability
of different leaf sizes just apriori or
just the probability of states a priority,
but generic,
and that's usually called a prior
probability in Bayesian inference.
So to do this kind of perceptual
inference right to figure out
what's most likely out in the world.
Right small leaf for big leaf we
use something called Bayes theorem,
which is where Bayesian inference
comes from and schematically it's just
posterior equals the likelihood times
the prior over just the probability
of the observations in general,
and I'll talk more about this
'cause this actually only makes
sense in the context of a model,
which I again will get to,
but so in more informal terms to just
be the probability of the state given
the observation is just the likelihood.
Dear observations given states,
times the probability of states,
then again over the probability of
operations and then you could cast this
again in terms of the specific example
here with the retinal image and leaf size.
But in this case,
what would probably happen is your
prior belief in your model would
be that small leaves are more
likely because you don't see a lot
of gigantic leaves out there,
and So what you'll perceive is a
small leaf that's up close that
is perceived to be close to you.
Just to give people a little bit
of a sense of where this sort of
equation comes from,
what's actually happening a
little more of an intuition.
What I can do is I can use what's
called the product rule in probability,
which just says the probability of
a given B times of probability of B,
Suzanne B, or just some events.
We can get something called
the joint probability,
so probability of A&B,
which is just the probability that
these events these two events will
happen together and what that means is
that so with this rule I can take this
this talk denominator here and I can
just turn this that's equivalent to
the joint probability of observations.
Given states.
And then this makes it intuitive using
these sorts of Venn diagrams which
are often used to teach probability
theory where we can just say all
situations where some of them oh is true,
is captured by this left circle,
whereas all possible situations
where some state S is true.
Or in this this right circle and there's
some overlap here in this blue area,
which is the probability of that observation
in that state happening together.
And So what we're doing when we're trying
to get posterior inferences were saying
conditional on this observation I just
got when we condition on that observation.
What we're doing is we're just getting rid of
that 'cause we know we're in this circle now,
and so then we're just saying, OK,
So what is the probability within, oh,
within this circle that we're going to ask,
right?
So we're just dividing this
part by the whole thing,
and that tells you the probability
that X will be true when you know
that you're in somewhere in oh,
so that's.
What posterior inferences about so now
when you're using this in a model of
the brain doing perceptual inference,
then the way we we usually talk about it
is there's hidden states of the world.
Ask that you don't know directly.
And those need to be inferred from the
sensory input or observations that you get.
Oh and then.
So I'm trying to yeah and then and
the the relationship between S&O is
specified by some internal model.
So we have parameters in the model
that says what the different possible
states are and how States and
observations are related.
And so that's captured like this.
This is exactly based here in like
I showed before.
The only difference is now we're
making it explicit that we're
conditioning on some model M,
so everything is just the same as before.
It's just me.
It's been stuck in there,
and so is your prior.
There's your posterior.
This term now probably about given M.
Often gets called the model evidence.
I mean,
again,
I'll talk about that in a second,
but it's just important to be clear that
this M is should always be in there,
right?
Ain't like it doesn't really make
sense to talk about probably
observations without saying that
that's what that's the case with.
Then some model,
but when you see it in practice in papers,
often the Ms left implicit any equations,
so it will just be that.
But the M is really always
assumed to be there, so I can.
I can give you an example
of why this is important.
So let's just say we have these data points.
These red and green data points here as well,
although the data.
Oh, and we have some likelihood
function and in this likelihood
function there's two different states,
so there's the red state
and the green state here.
And these circles just denote
probability distributions, right?
So so under the red state,
these are the probabilities of those
observations and the green state.
There's high probability of these green dots,
and so the model evidence then
just adds up the probability of
observations under each state.
So it's basically that probability
plus that probability.
And as you can see,
this model does pretty good because
between these two distributions most
of the data points are pretty well.
Covered,
which means that the model
evidence on a whole is good,
and the model is good at accounting
for observations generally across
the states that it has in it.
In contrast,
if you were to have a likelihood
function or a model where the two
probability distributions look like
that for the different states,
this doesn't cover the data well at all,
and so that means that the
model evidence will be low,
and so we can call that a bad model.
Now just to give you a sense of the
actual how you'd actually carry out,
kind of like a concrete
mathematical example of this.
This is kind of a fun one.
I'd just like to show in the past,
it's just a kind of standard
perceptual mode conceptual illusion,
but a perceptual effect that's
easy to show if you just look at
this kind of middle circle here
and just kind of ask yourself,
you know, does it look?
You know, just just naturally,
does it look like it's
popping out like it's convex?
Or does it look like it's
popping in like it's concave?
Here,
'cause those are the different
possible causes out in the world.
And,
well,
you'll notice is at most people
anyway perceive this as looking
like it's popping outward when
you do experiments on this,
whereas if you just flip it around now.
Most people instead tend to
perceive this as being concave,
that it's popping in.
So the question is,
I mean the just the image on the retina
is equally consistent with both of those.
So why does this effect happen?
Well,
it's because it looks like
we've evolved to have a model.
It just assumes that light
tends to come from above,
and if light comes from above
then you're only going to get the
shadow on the top if it's concave.
And so we can model this.
You know,
using Bayes theorem just
like in any other case,
to infer which one of these things it is,
and that's what your brain
is doing implicitly here.
And if we would actually carry
this out mathematically,
as a simple kind of worked example,
we can just say the probability
that you see that shadow pattern,
given that it's concave and my
model assumes that light comes from
above versus convex is point 9.1,
so there's kind of a precise
distribution here that it's worth.
Favors one over the other,
whereas we can just say from the start.
Maybe there's a fairly flat,
imprecise prior belief that concave
and convex are pretty similar.
Probability of being very sorry
and then to calculate the joint
probability we just multiply
across the rows and so we get.
Is to do. Is 2 joint probabilities.
We can add those together to get
the the model evidence or what's
also called the marginal likelihood,
just like in the example with the
red and green dots I showed up and
then we can just divide each of the
joint probabilities by the model
evidence here and we get our posterior,
which would say it's probability of
.88 that it's concave and probably .12.
That's convex and so we perceive
it as being concave.
So you can see actually once you put
it in concrete you know mathematical
terms with numbers that it's not as.
Complicated as maybe or daunting
is maybe it can seem to.
Some people don't have a lot of
mathematical background is over with,
but I keep point here is is that
more precise sensory input.
So in other words,
having a really precise likelihood like
point 9.1 tends to lead posteriors
to be more similar to the likelihood,
right?
So it's a lot more likely you're
going to get a distribution
here that's similar to that,
whereas if you had more precise
prior beliefs if instead say you're
started out with a strong belief
that it was going to be convex,
that would kind of pull the
posterior distribution.
To look a little bit more like the prior
prior beliefs is this kind of competition.
Between the prior and the likelihood,
and determining what you end up
perceiving based on your posterior,
so two other just kind of quick.
Just some mathematical kind of fact
that I want to highlight for people so
they can understand a little better what
what's going on in some of the actual
active inference specific equations.
So one this is just a just a fact
about probabilities is that when
you take the natural log so Ln of a
probability it converts multiplication
operations to addition.
So for instance.
If I take this probability B&B and
it's and it's a product of those two,
then if I just log log all of these,
then all of a sudden instead of it
being this being multiplied.
It's just added,
so it's just the log probability of.
That condition would be given
A plus a instead of the most,
and so the product of two and it also
similarly converts division subtraction.
So if I have the joint
divided by probability pay,
then if I log it then it's it's one
subtracted from the other instead
and then we can easily convert
them back to normal probabilities
just by exponentiating them,
which just means I take E
which is a Euler's number.
It's just a constant.
And I and I and I then exponentiate
it's OE to the LNP given a it just
gives you back the probability of a.
So we can get back to normal
probabilities whenever you want.
It's just sometimes more convenient to
work with log probabilities because
addition and subtraction tend to
be easier and in some cases than
multiplication indovision the other
quick thing to note is that adding some
constant two operations on probabilities
just doesn't change anything.
So the probability of a plus.
The probability of not a equals one.
If it's true,
then probably given C and not a given.
See also equal 1.
So just conditioning or constant
doesn't change anything,
and the same thing with multiplication,
and so you'll see that will
also happen sometimes.
OK, so after all of that and get active
inference and so so if we start from
perceptual inference where we're just asking,
so what's going on in the world?
Just as another example,
let's say there's some person out in
the world and you're trying to infer,
say, their emotional state,
or they're kind of disposition towards you.
You know, are they neutral?
Or are they a nice person?
Or they I mean person based on sensory input?
You'll try to infer some
model of that were active.
Active inference comes in,
you know, let's say in this sort
of social interaction example.
Instead, you're trying to infer.
What should I do?
This couple's perception and action,
together where what you want to
know is where you want to predict
is what's the probability that
I'll get some new sensory input,
given that I choose action a
versus action being, you know.
So maybe I'm considering acting
friendly and in my model,
let's say that my prediction is
that I think friendly is going
to lead to sensory input,
then being this kind of genuine,
you know nice smile.
And then I'll say yes,
'cause that's consistent with my preferences.
Whereas instead let's say in my model
it's more likely that I'm going
to get some kind of judgmental.
You know.
Potentially ingenuine smile,
you know something like that,
which I don't prefer.
Then I'll say no, right?
So it's probabilistic inference,
just like in perception.
But now we're inferring probabilities
over different possible actions.
You can already see in terms of like,
potential psychiatric relevance,
how this kind of thing can lead
to maladaptive sort of self
reinforcing loops where you know
if you choose not to act friendly.
For example,
the other person may just think you
don't like them and then you just
end up continuing to lack the social
support or continue to lack social support.
And then you just end up unintentionally
promoting the act without sort
of social rejection that you're
trying to avoid that you fear.
So last point real quick on the
kind of conceptual side is that
it's important to recognize that
active inference and perceptual
inference doesn't just apply to
interactions with the outside world.
It also involves interactions
between your brain and your body,
so you have to infer you
know what's going on.
In my body, you know what is my heart rage,
you know, I like how fast am I breathing?
You know what's what's going on
in my stomach, things like that.
And then you also have to
infer something like action.
You have to say how?
How should I change?
Going on in my body to meet expected
future demands in a situation.
And obviously you're not
typically doing this consciously,
but your brain somewhere in there
is still having to make decisions.
Should I increase heart rate right now?
Because say,
I expect I'm going to have to run away
in a second and I'm going to need
more blood flow to certain parts of the body.
That's also part of active inference.
So, so the general idea here is
that these these basic ideas that
have introduced now conceptually,
I think it was a little bit of the
basic mathematics active inference.
The formalism just just formalizes that.
It makes it quantitative and allows
us to simulate those processes and
provides parameters in these models
that allow us to capture individual
differences in people's behavior
in these sorts of circumstances and
an active inference models this.
This version of them.
This decision making version
are based on what are called,
partially observable.
Markov decision processes or comps?
And more or less they just included
everything we've already talked about,
except for the bad time.
So you have hidden states of the world,
but they evolve overtime.
Otherwise,
it's just like before the agent doesn't
have full knowledge of the environment.
Instead,
it has to infer the hidden states of
the world based on the incomplete
information against observations.
So really it's the same as
we've talked about.
It just adds like evolved.
How states evolve overtime.
Now at this point I'm gonna have
to kind of load you up with a fair
amount of like terms and notation and
some other mathematical constructs,
so it's gonna bear with me a
little bit and this this ultimately
I promise will be helpful.
So we're calling,
oh,
we're using over observations
like we talked about.
These are also often called outcomes,
but observations and outcomes
are interchangeable.
As will be our hidden states that
could be inferred from observations
as we've talked about already.
Hi here, so this is the next new thing.
Corresponds to policies and policies are
just sequences of actions that you could.
Where each action is denoted by you.
So for example,
let's say I start and this
starts state right here,
and my goal is to get to this star down here,
which they has a reward of plus
10 or its preferred plus ten and
all other states just have a - 1.
So what you want to find is whatever
policy is going to get you to the
goal with the fewest number of steps.
So policy one here.
Let's say that just involves a sequence
of 10 actions just going down into the right,
whereas policy 2.
As 14 actions, and so you're going
to get more negative ones,
so policy one will be better.
But the point is is that policies
are sequences of actions,
they're not.
This is a little bit different
than in reinforcement learning.
If people are familiar where policy
corresponds to the mappings between
States and actions, whereas these are,
these are trajectories.
And so under active inference,
because the whole goal is to choose a policy,
we actually evaluate everything
probabilistically under policies.
So for instance,
our prior now is the probability
of states given a policy.
So this data expect to be in.
If I choose a given policy,
whereas our likelihood now
is also conditional policies.
So the observations I expect if I'm in
this state and I'm following this policy.
And our posterior same thing,
it's just the state.
I infer that I am in after
making this observation.
If I'm following this policy
and that's often the thing that
we want to figure out, right?
We're trying to do inference
to get a posterior.
So this is all we need to perform
base theorem just like before,
except for a policy that is kind
of what's taking the place of
our of the model variable M.
In the version I showed you before.
Which policy can be kind of a
considered its own model of a sequence
of sequence of States and actions,
which again will cover an
event so the next thing,
two things that are really
important to understand and active
inference are what's called the.
Active, posterior or observation.
So the observations that I expect
if I choose a policy and this
thing probability of oh given C,
and this is a very unique
component of active inference,
where this distribution
is encoding preferences.
So in other words,
the observations that I find rewarding are
encoded as having a higher probability.
So higher probability observation in
this distribution just equals more thing.
So this is a unique element in
active inference where you're
using something with the form.
Of a probability distribution
to encode reward.
And then the next,
the next thing that's important to
understand sort of pulmonary is
that often in real world cases doing
exact Bayesian inference becomes intractable.
The problem just becomes basically
it would like either take too long
to solve or it's just not not solvable.
Exactly because the mathematics
get really complex and and So what
we need is to use approximation
techniques to approximate the
distribution that we want to know,
and so Q denotes the sort of
approximate distributions we're
trying to get as close as we can to
the sort of the true probability
distribution that we're looking for.
So,
for instance,
queues given pie will be the
approximate posterior over states,
which is just a distribution.
You'll notice it's not.
It doesn't include oh in it,
because it's just a distribution
over states that we're trying to
kind of wiggle around to figure
out how to get it as close to
the true posterior as possible.
And then Q evo given pie would be the
approximate prediction about what
I will serve if I choose the given policy.
So now the way this ends up contributing
to decision making processes as well.
Go into and.
A little bit depends a lot on
this construct called the this
mathematical construct called the
callback library divergance or KL
divergent and what this is just
saying is the KL divergences between
some probability distribution X and
some other probability distribution
of Y and so here's smaller values
just indicate the two distributions
are more similar and so for example,
the closer we can get. The.
Predicted observations given all of these
to match our preferred observations
to smaller that KL Divergent will be,
which again is very important
for trying to choose the policies
that will give you what you want,
and this is just a the actual equation
for calculating and the KL divergent,
which is ultimately not not very complicated,
but but so just as an example,
let's say that we're considering the
observations that we expect given we
choose policy one and we're comparing that
with the observations that we prefer.
Let's say this distributions look like this,
so here are the red distribution
according to.
Which is our expected observations.
If we choose policy one,
you can see it's not really that
similar to the blue distribution
which encodes the what we want.
So the KL divergent there will be large,
whereas we consider policy too.
Then the observations expect that
under policy to overlap quite a bit
more with our preferred operations
than the KL Divergent will be
small and all is being equal.
We can stop that policy.
Two will be better 'cause it's getting.
Closer to what we want so it
gets the skip the Gold Star.
OK,
so another couple of slides here
to talk about things that we still
need to know so one is is that
and this is something that can.
Obviously it's awesome is it can be the
most confusing when you first learn English.
Two different types of time
that need to be separated.
First type of time with.
You know with this variable Tau and that
denotes the time about which I have a belief.
So to make this blue air,
so consider that I believe right now
that I'm in my car right now, right?
So that could be Tau.
But I also can believe right now that I
was in my kitchen 10 minutes ago, right?
So that's a belief I have now,
but it's about some previous time point.
So town minus one.
And I can also believe right
now that I will be at work.
In 20 minutes, right?
So that could beat out plus
one some future time.
So it is I can have beliefs
about each of these time points.
You know right now,
whereas T corresponds to the time
at which I get a new observation.
So for example, after I turn on a light,
you know at Tau I might now update my belief.
About where I was five minutes ago, right?
If I turned on the light and I know,
so hey,
I'm in the kitchen and I've been in
the kitchen for the last five minutes.
So so T is when I get an observation.
But then when I get that observation,
I can update my belief about
the states at all tabs.
So hopefully hopefully that helps clarify it.
I know I know that one can be
a little bit tricky at first.
Let's say yes like I said,
I can update milli by belief about
states at all towels with each
new observation at each team.
And then the the final thing is
that in models often we need to
have more than one set of States
and more than one set of output
modalities or observation modalities.
So, for instance, I might need one
set of states corresponding to say,
like beliefs about the identity of an object,
but a different set subset of.
States that impose beliefs
about the location of an object,
and so we call those different
state factors and for observation
or output modalities.
We might have one set of outcomes for
vision and another set of outcomes or
observations for audition or hearing,
and then the final thing that I
want to make sure is clear before
we really go into the kind of nuts
and bolts of active inference is
this distinction between what's
called the generative process and
what's called the generative model.
Uhm,
and the idea here is that the the
generative process describes the the
kind of actual objects and events they're
out in the world that are generating
the observations that the agent is getting,
whereas the generative model is
what's actually in your brain,
right?
It's your beliefs about the
generative process and how it
will be affected by action,
and maybe that model is accurate.
And maybe it's not,
and so that's usually expressed
as just encoding a probability of
each combination of observations,
States and policies.
OK,
so having introduced all of
those preliminaries,
we can now dive into trying to understand
this actual graphical model depiction
of what active inference really is.
So here for people that are
unfamiliar with graphical models,
arrows denote conditional dependencies.
So this is saying,
for instance what this observation is
depends on what the state of the world is,
and so on and so forth.
Circles here correspond to random variables
that are updated during inference.
So in other words,
with each observation we update the belief,
say in this circle about the states were
in and about what the best policies are.
The squares instead indicate the fixed
parameters of the model that encode,
for instance,
priors and likelihoods, and so forth.
So I guess that this can always,
you know,
for for people who are new in this area,
this can often be a bit daunting,
so I'm going to walk through this
kind of very slowly step by step.
And just to note from the beginning
'cause I'm going to kind of look at,
I'm going to work through this bottom
part first before adding action.
So this bottom part really just
corresponds to perception,
whereas this.
Part where the states overlap corresponds
to the action selection component,
so we're going to go through
perception first,
and to do this because I really want
to show how this can be applied
to the sort of behavioral tasks
you can do it in an experiment.
We're actually going to build a
model of a particular sort of fairly
straightforward to our motorbike.
2 armed bandit tasks,
or slot machine tasks with a
little bit of A twist to it.
So what we're going to say,
as the agent can pick one of these
two slot machines in each trial.
And the and one of them will payout $4,
whereas the other one will be out at $0.00.
But the agent doesn't know which is which.
So what could happen is it can either
choose to take a guess immediately
and just kind of hope for the best,
or it can choose to ask for a hint first.
It'll give it some information
about which one is more likely to
be best or more likely to pay out,
but if it chooses to take the hint,
then it can only win $2.
If it gets it right instead of four, Sir.
Is this kind of information seeking?
Versus risky reward seeking aspect to the
task and and so for this just as an example,
we would need two hidden state factors,
right?
We would need some context state factor
where the agent has beliefs about
whether the left machine versus the
rag machine is more likely to payout.
And then hit the second hidden state.
Factor can be beliefs about our choices,
right?
So starting in the start state or the
state of asking for a hint or the state
of choosing the left versus right machine.
So this is kind of the way you could set
it up and then the output modalities one
could be observations about the hint,
right?
So you might not get a hint if you
just make a guess immediately.
Or you could observe the hint that
the left machine is more likely,
or that the right machine is more
likely and the other upper modality.
Would just be observing a win or
a loss you know.
Did you get a payout at the
end or did you not?
OK, so using that model will start
out just looking at static and
dynamic perception in these models,
so starting out just with the
simplest little part at the
bottom of the graphical model,
we just have our observation of
and we have our interstates pass,
so those you already know and now D here
is just the term that we're giving for
prior beliefs for the first time steps.
This is just saying.
This is your prior about
what the states will be,
whereas a is just a matrix that
encodes the likelihood mapping.
So now and then the equation
that we use is essentially just.
A version of Base theorem so in log space,
right where we're adding the prior to
the likelihood because it's in log space.
Otherwise would be multiplying if
it wasn't so this is equivalent
essentially to base theorem.
The only thing to note is this Sigma.
Here is what's called a softmax
operator and it essentially just
takes the values and distribution and
then just transforms them back into
a proper probability distribution
that's proportional to the numbers
in the original distribution.
So it's just.
Transforms it into proportionally
non negative value system to 1.
So for the slot machine task,
you know we might just say OK
are probably for the beginning,
but the left machine or the right
machine is better is just point 5.5, right?
We have no idea.
So both machines are equally likely
and then it could get the observation
of just a one and a zero and we can
just say that this top one corresponds
to getting the hint that the left
machine is right and if we get that
then if RR likelihood they might look
like this and if it does that will happen.
Is this top one here?
Selected this robot upgrade.
And So what that says is that the
probability that the left machine is
right is .9 whereas the right is .2.
So what that means is the hint
should make you believe that the
left machine is quite a bit more likely.
So then what we do is we just just do
the equation based on those things,
right?
So we can just work it through
given those numbers following
that equation and we get that the
probability that the left machine
is going to payout is pointing to.
So that's the way it would work in practice.
So now moving on to dynamic perception,
all we're doing is we're just
taking the static perception thing.
I just showed you and just
stringing a few of them together.
So now there are states at Tau equals
one cow equals two to equals three,
and the only other thing we're
adding is these bees here.
And so B is just a matrix that includes
the probability of state transitions.
So it's the probability that
you're going to have a certain
state at T out to equals two.
You know,
given what the stat,
what your belief is about.
The state attorney.
This one so you could think about
these is just providing prior beliefs
based on the previous based on
beliefs of the previous time step.
And the equations for solving this are
are being basically just equivalent
to the ones solving the other,
solving the previous equation.
I showed you just were adding time,
but here the likelihood exactly the
same and the prior here it includes D.
The only thing that we're
adding is this b * S Here,
which is basically giving you know,
providing beliefs about your allowing
beliefs about future states to
also help you infer what you're.
With your current status and then four
and this is for when Tau equals one.
When you're inferring states for towel
greater than one same kind of thing,
except her prior is just beliefs
about the past through B,
and beliefs about the future through B,
and then your likelihood still
just the same thing.
So again, it's just Bayesian inference.
Anything to note.
Sometimes this can be a
little confusing to people.
Why you include this prior belief
from the future when you're trying
to infer the state that you're in
and the reason for this is because
because you're having pulled beliefs
about all time points at each,
you know with each new observation,
then you can.
This allows you to kind of
retrospectively update beliefs about
earlier time points when you get an
observation at later time points,
and then I'll give specific examples of this,
but so now.
That we have this dynamic
perception structure in place.
One really key thing in active
inference is the use of water
called variational message passing
algorithms for solving this,
because remember exact Bayesian
inference is often intractable,
so we have to have some kind
of approximate way of doing it
or getting as close as we can.
And So what?
I'll show you the equations that
I showed before implement this
variational message passing
scheme that as I'll show later,
actually I can provide.
And I sort of theory about message
passing between neurons in the brain as well.
And I should say this is a slightly
modified version of variational
message passing called marginal
message passing that it's just
kind of a refined version that
has some additional benefits to
it that I won't talk about.
But we can talk about later if you want,
but it's just an efficient way to
do approximate posterior inference
on one of these graphical models,
and this is based on the way
variational message passing works is.
It's based on minimizing.
Quantity called variational free energy,
which is a kind of central
component of active inference,
but I'm going to kind of show you
give you an intuition about how the
dynamics work with message passing,
and then I'll introduce variational
for energy formally.
So basically,
the way this works is you'd
start by just giving the,
giving the agents the first observation.
So we'll just say that's a one,
and then we pick some particular state
that we want to infer a posterior over.
And what we do is we just pass messages.
We just pass that message from the past.
We passed that message from the future,
and we pass that message up from the
observation through the likelihood,
and then if we do a softmax on that,
then to turn it back into a.
Proper probability distribution.
Then we get an updated belief.
QoS approximate posterior
overstates for tally equals one.
You know if we actually worked
that out and then we would end
up with something like that.
And then if we stopped to get Q
then we end up with 25.45 or or.
So that's our new approximate
posterior leaflet states Italian.
But now what we do is we now
just picked this next one.
Pick, you know, pick the next picks the next
state that we want to infer a belief over.
Now for a future time point past the
messages here accusing our new queues here.
Same thing we passed those
three messages together.
We get our Q s = 2 and then we did
the same thing here and we get our new
updated posterior for Q for for towels.
For us stands at High Street
and then the idea is we just.
Iterate this process over and over again,
so past messages give you
posteriors past messages,
getting posteriors and so forth,
and then eventually these posteriors
will converge on this table.
Values with enough rounds
of passing messages.
So if we assume that the
states have converged,
now we just add the new observations,
so now this becomes A01 and then
we just do the whole thing again.
We just repeat them until the police.
Exact same thing for three and so far,
and so the observations we have
this final kind of posterior belief
about all states at all times.
And you can see that in some problems this
is just showing the kind of simulated
dynamics of something like this.
So here I have 6 traces,
so this is essentially one trace per.
Each of these states there's two
possible states that can be in right?
So we showed this example.
Here where you got up.
So right, so there's two possible
states where you can be in or Italian.
This one, two states for two,
two states for training.
This traveling is just. Actor. Uh.
This is just showing one trace for
each of those answers, so there's two
two choices for each of the police.
This case, it converges pretty
fast to a precise belief in one of
the States and not in the other,
or the probability becomes zero.
But there can also be harder and for its
problems where it takes a long time,
it takes a lot of iterations
before the thing will converge.
There's quite a bit more evidence
accumulation that's necessary.
And the when the actual observations
that you're getting are more noisy
or the likelihood is less precise.
So you know, as I mentioned before,
briefly, one nice thing about this
sort of scheme and what's proposed
in active inference is that message
passing also represents a way to
think about the messages passed
between neurons in the brain.
So and you can formulate it in terms
of minimizing a prediction error,
just like in you know something
like predictive coding or other
Bayesian brain type theories,
and the way that you can do that
and the details of this don't
really matter a ton right now,
but you can just move.
You can just set this to zero by moving us
over here and getting rid of the software.
Facts and then you have something
that looks like a prediction error
where this equation is driven to
find values for this prediction
area that's close to zero,
and this is essentially just the
generative model after an observation.
Subtracted from your previous
approximate listeria overstates,
so you're just trying to
get these to match up,
which is exactly what was happening
when I was showing you that kind
of example message passing,
and then what you can do to make
this sort of more other than this
prediction error is you can update
this value the you can just ask for
like a memory in voltage if you're
trying to be specifically biologically
motivated here where we just
corresponds to the log of log events.
And then when you smoke packs
that you end up with something,
it's like a normalized pirate, right?
So then you get predictions about
dynamics in number and in firing rates.
And so so just to to walk you
through again what this will look
like in a in a concrete example,
let's go back to the slot machine
task and let's just say that our
transition beliefs here be just
corresponds to this identity matrix.
So this just passes if the left
machine is better at time .2,
then it's still going to be
better at time .3 for example.
With probability one.
So if I do that then then what I
can do is I can just plot with
each new observations that can
simulate how beliefs will change.
So the way to to read these sorts of
groups which you'll see all the time
in papers in the active inference
literature is you have two rows here.
So beliefs that the left machine
is better or the
right machine is better for states
that Alec was one same thing,
beliefs about what's better at Alec
was two and same thing for tackles.
Hurry up and then each of these columns
is going to be when you get a new
observation and darker corresponds to
a higher probability and I'm going to
put these little cyan dots as well.
Little note the true observations
or sometimes true states or actions.
So if I just say it the first time point,
the agent gets no lobster.
Vacation doesn't perceive any hit,
then this just stays Gray.
Essentially the probabilities for
all time points or just point 5.5.
Just completely imprecise.
But when I get the left hand observation.
Now all of a sudden my beliefs
pretty quickly.
It doesn't take very many iterations
before the thing infers that the
left machine is better remembered,
darker is higher probability here,
and at time .3 it doesn't get
any additional information,
but this just carries over because the
transition beliefs are fully precise.
So in other words,
states that time to stay the
same at times three.
But the important thing to notice here
is that it doesn't just update its
beliefs about which one is better.
For Tau equals 2.
It also had beliefs immediately,
retrospectively that the left machine
was better in the past and that it's
still going to be better in the future.
So this is this kind of retrospective
inference aspect that can be really
useful in some modeling context.
And and so you can use this to generate
neural predictions in a particular way,
so it's important to notice that
in the normal process theory,
each of these rules just correspond
to that thing kind of convergence.
If you weren't ready for and you
can recruit those sort of things
about firing rates.
Atlanta generate neural predictions
for it's it's in EG studies.
All you do is you take the rate
of change in these and you get
what look like well look like
predicted event related potentials.
So basically how quickly beliefs
change is what generates stronger
or weaker ERP's it would be
predicted say like in any EG study.
Come now if instead we change
this for instance,
and we make the transition
beliefs less precise.
So now this is just saying if the
left machine is better at time two,
it'll still be a better time .3
and only with probability .7.
So that goes to left but only 1.7 now.
I'm so now if I do the exact same
thing and it deserves the hint,
then now we'll see is very precise
believes that the left machine is better
for states, that's how equals two,
but.
It's not very confident anymore that
that was true at how equals one,
or that it will still be true and
how equals three because it doesn't
have this precise belief that
states stay the same over time.
And so in this case, if we do it,
if we use it to try to predict something
about neural responses will get. Is you know,
like a are predicted firing rates.
They look like this so much less so
much much lower convergence for this.
These are traces for time one and time three.
We get a very different predicted
pattern of ERP is it's the scale is
different here than on the previous one,
but these are much smaller so so that's
one way that we can make predictions.
And there is evidence that you know
this way of generating predictive ERP
is actually may map onto something
really might be useful.
Uhm, there's a few studies.
For instance,
you know this is a one that we did.
This is another one that I'm
being Christopher White dead,
so the other one.
The other authors on the teacher or real.
But each of these either shows
something like that.
If you simulate a task,
you will generate simulated TRPS
that already look like what's been
observed empirically or in this one.
Like I'll show you later,
the parameters in the model actually
do predict things about differences
in empirically measured ERP's.
Just as a couple of examples here,
for instance.
This is an empirically observed
ERP inattentional blindness task,
and this is the simulated ERP.
When you build a model of this task so you
can see that that it matches pretty well,
whereas this is just showing this
is from one of our studies showing
that a parameter that essentially
reflects differences in the the
precision of the likelihood mapping
predicts the more precise it is,
the more strongly the more
strongly RP's are in in posterior,
like parietal cortex,
and then this is a similar.
Just another example.
Or if you build a model of an
oddball task then you can reproduce.
Things like like mismatch,
negativities or P3 hundreds
that are observed empirically.
OK,
so now that we've gotten through
these examples.
Message passing I mentioned that
this is all based on minimizing this
concept of variational free energy,
and the idea with variational free
energy formally is you just start
with some generative model that you
know right which corresponds to.
Like I said,
this joint probability or the likelihood
in the prior if you separate it out.
And then you introduce what's an
initially an arbitrary distribution.
So our approximate distribution,
QS,
and we want to try to use that to
approximate the true posterior that we want.
So PS, given no.
And the way we do that is with
this variational free energy.
A measure which is just this or
this is one way of expressing it
and what you can see here if you
if you kind of decompose it is
that this is just the approximate
posterior over the true posterior.
So basically,
the closer it follows that
the closer this gets to this,
the more this whole thing will
just equal 1 right?
If this is equivalent to that, this will
just be one and the log of one is 0.
Uhm, so in other words F will just
get smaller and smaller the closer
QoS gets to the true posterior.
And when this is zero then
you're just left with just.
Stab the model evidence again.
M is implicit here,
but this is just the the the the model out.
It's the probability of observations
in general given a model.
So it's the difference between the
approximate and true posterior,
and this just kind of notes.
Generally, how surprising is
an observation under my model?
So as I said, as F approaches 0,
QS becomes more similar to the
posterior distribution or the yeah,
the poster true posterior,
and so we can just kind of wiggle
around and try out different values
of QoS and find one that find the one
that gives us the lowest value of F.
Uh. And just as a kind of numerical
kind of concrete example of this,
let's we can take this version of
exact Bayesian inference where I've
just arbitrarily said the prior is
.5 and .5 the likelihoods 28.2.
And then I can calculate it through
and get the posterior probability
under each observation.
But this part right here going
from the joint to the marginal
or to the model evidence.
This is what's often intractable
in real world cases.
This just gets too difficult
as a computation,
so instead we can use variational for
energy minimization to solve this in a
way that is computationally tractable,
and to do that we can just start by
setting QoS equal to what we know,
which is our prior so point 5.5.
And then all we do is we just
try out different values of QoS.
So say we initially start out with point 5.5.
If we calculate F out we get a value of .916.
So now let's just say we move it.
Now we try QS equals point 6.4.
We calculate it out and now
the value is .798.
So we notice it got smaller
if we do it a second time,
move it 7.3 F, get smaller again.
If we move it two point 8.2.
Which is remember the true posterior.
If you could do exact inference,
then F becomes .693 and so we know that.
So this is the perfect match,
but to confirm it,
let's say we try out point 9.1.
Now we can see that F gets bigger again,
so we know that this is the minimum,
and so this is our.
This is our approximation to the
true distribution true posterior,
which is exactly true in this case,
but won't always be exactly true in
those cases more complicated cases.
One other way that that's useful
to express variational free energy,
and that's often shown in that that does
provide a nice intuition for the way
this is working in some cases is by,
by decomposing it into these two terms.
So one this is just the KL divergent, right?
The difference between QS and PS.
And this just says essentially
how much do beliefs change.
So when I get a new observation,
how big is that difference?
How much does your prior belief?
Change to your posterior belief.
And we can call that complexity and so,
whereas this term here it's just
the probability of observations
given states under your model,
and so that's just saying the
the larger that is,
the more accurate your predictions
were essentially right.
The probability of the observations
was higher under your model,
and so we can call that accuracy.
Equivalently, you could just flip
the sign of this and you could call
it complexity plus prediction error.
It's essentially the same thing so,
but what this means is lower F
values are going to be the ones
that move posteriors as little
as possible for prior beliefs,
while also maximizing accuracy.
So essentially you're trying to
find the most accurate beliefs that
change her beliefs in the most
parsimonious way that you don't change
your beliefs more than necessary.
And so convergence to QoS and the
message passing schemes we talked about
before then can also in this case,
because that message passing minimizes
F can also in this way be considered
prediction or minimization as well,
because F is essentially a
prediction error plus complexity.
And an important thing,
just to just to notice here briefly
if you're not familiar with it,
is you know why.
Why worry about this complexity thing at all?
Why not just maximize accuracy?
And the reason is there's always a
concern about spell overfitting, right?
So like these, are my data points,
then you could see each time you make
the model more complex you can get it to.
Set the dots better,
but if you make the Model 2 complicated
then you cross every dot perfectly,
but this one will make terrible
predictions about future data points,
whereas this one,
this kind of medium complexity
one is probably best, right?
So it's the one that will.
That will give you.
Probably the best predictions
about future observations,
so you don't want to change your beliefs to
make your model more complex than necessary,
essentially.
OK,
so that's the kind of perceptual
inference part in the way that active
inference does approximate posterior
approximate Bayesian inference and
perception by minimizing variation.
Free energy again using message passing.
So now we can move on to the
actual decision making component.
Here all we're doing is we're just taking
what we already looked at and we're
adding on the influence of policies.
So what's happening here is
we're now just saying that.
So Pi here says, OK,
what are all the different possible
sequences of actions I can take?
So,
for example,
taking the hint and then choosing
the left machine or immediately
choosing allow machine, etc.
A pipe does in this case is is.
It just specifies what
the state transitions are.
So each politician tells a different
sequence of transitions between states.
Since transitioning from one policy
might say transition from the start
state to the take the hit state,
whereas another might say
transition from the start.
Is choosing the left machine state
right away and so you could think about
each of these as just doing message
passing in parallel for each policy.
So you estimate,
for example,
what the posters over states
are given policy one which
has transitioned. You know,
type one here from different types here,
whereas the policy to be doing lots
of passing for policy that's on here,
but then transition three here.
Instead transition to their two policies.
Here is estimating it in parallel.
So now if we look at what this
would look like in practice,
for instance, this would be.
This could be like a transition
matrix that would correspond
to one kind of action, right?
So this might say no matter what state
I'm in, transition to state three,
which would be choose the left machine right?
So that could be one action,
just choose lock machine.
B1B matrix right? There might be like
B1 and what I showed you a second ago,
whereas choose the right machine would say
for many state moved down to state four
which would be choose the right machine
and this one could be take the hint.
Same way, so it's just to give you
kind of a practical sense of what
these would look like numerically.
So now in this graph policies
pie depend on this thing.
Gee, which we haven't talked about yet,
but that's the expected free energy,
and that's what's going to play
the role in deciding what policy
is better than what other policy.
So it's essentially saying how
good is each policy.
And C as we talked about is that
preference distribution, right?
So it's telling you what's
preferred over what else and so.
All cities with the lower.com.
He depends on 3rd and see talk about gene.
Also has another component in
it that also drives the agent
to maximize information game.
And I'll show that in a second when we walk
through what the actual equation is for G.
And then the one other thing that
I did want to show just again.
So you have like a concrete numerical
sense in your head of what it looks like.
This is what like a C matrix would look like.
What preference distributions
would look like?
So this would basically say at time
one I have no preference for observing
a star to lose or win at time two.
I have a preference, I have a negative.
Preference for losing a strong
positive preference for winning.
So four 'cause you'd win, you know $4.
Remember,
but if you take the hint first and so
you don't observe the win until time three,
then you a week or preference
of two instead of four,
because you only win $2.00,
right?
So this just drives the agent to want
to observe a win sooner rather than later.
And then in practice what we do is we just
softmax that we softmax each column to turn
it into a proper probability distribution,
and then we log it.
So that's what actually goes
into the equations.
And I want you to keep this this in mind
here because the bigger that number is,
the more precise the
preference distribution is,
and we can use that as a parameter
to fit for individual participants
in a study that essentially can tell
people how information seeking versus
how kind of risk seeking they are.
But again, I'll come back to that.
OK, So what is expected for energy?
So if we remember here,
variational for energy pertains
to current observations.
So you have to know what
the observation is right?
So that can only really apply to the present,
whereas decision making requires predicting
future observations under a policy,
right?
You have to say which which
observations do I expect,
given that I do you know thing
one versus thing 2.
And that's where Q outcomes in here
where oh is treated as a random variable
that can take on different values,
and so you're trying to predict what
values are more likely than what others.
Calculating the expected free energy
has to do with these predicted future
observations under each policy,
and then we can use G to score which
policies are better than others
based on the observations we expect.
Now the the active inference later
shows a bunch of different ways of
decomposing expected free energy.
11 useful one is this what's
called the epistemic value and
pragmatic value decomposition.
Now again, I mean it's gonna look kind
of complicated and opaque at first,
but really all this is saying is what's
the difference between my beliefs about
states after I get an observation?
And what were my beliefs about
states before I got that observation?
So essentially, the bigger that number is,
the more information that I got,
the more my beliefs changed about
the states that I'm in.
Whereas this is just the probability
of the observations you prefer.
Right, so this is just saying the
expected probability that I'm
going to get my preferred outcomes,
and so as each of these get bigger
because they're both negative,
then she gets smaller.
So what that means is just that agents
will select policies that maximize
information gain and that simultaneously
maximize the probability of reward,
and these will kind of be weighted against
each other during decision making.
Another really common decomposition
that's worth showing is what's called
the risk plus ambiguity decomposition.
And then this one again.
It's the same kind of idea.
You're just saying what's the KL divergent?
So how different is the?
Probability of observations you
expect under a policy and your
preferred observations.
This is literally just what I showed
you guys before when I introduced
the Cal Divergents and you want
to get this divergance to be as
small as possible right?
Hip?
The policy that's going to get
the observations that look the
most like preferred observations.
And then the ambiguity term is
just another way of kind of
expressing the epistemic value.
So this is just saying how informative
do I expect an observation will
be if I move to a certain state,
you can think about this just
in terms of that. Eventually,
the precision of the likelihood here,
so you could see for for state
one in this example,
probability of observation.
One is .9.
The probability of observation, 2 is .1.
So if you observe one of
those two observations,
you're going to gain a lot of information
about whether you're in state.
Whereas if you move to state two,
then it's point 5.5,
which means that no matter what you observe,
you're not really going to gain
any information about whether
you're in state two.
And so this one is the lesson.
Biggie was one,
which means that so it's lower
entropy or higher precision,
and so the agent is going to be driven
to move to state one because it's
less ambiguous and more information.
You get more information.
So the the reason why I expected
for energy is interesting.
You know using it as opposed to just
maximizing reward as opposed to
minimizing free energy when you're
expected for energy when you're making
decisions is 'cause it helps a lot in
arbitrating decisions that involve
this kind of explorer toy dilemma.
Where where you need to decide whether you're
going to seek information first or just,
you know, take a risk and go forward.
Now, even if you're have some uncertainty.
So what typically happens in in
tasks is the policy selection only
initially be driven by the ambiguity
terminal B information seeking,
whereas later once the agents
confident then it can just leverage
that information to pick the policy
that will minimize this divergent.
So maximize reward.
Uhm?
So just to kind of summarize,
each policy entails a different
set of state transitions.
Those in turn predicts different
sequences of observations,
and those jointly allow a valuation of
the expected free energy for each policy
based on they predicted States and
therefore they predicted observations
and which ones you prefer over others.
Now,
if you I won't walk into this and in
detail it doesn't really matter that much.
But when you translate expected for
energy expected for energy equation,
enter the matrix notation.
It just looks like this where this
is just the risk term.
Again,
the difference between expected observations,
which is just the likelihood times
the states you're in,
minus the your preferred observations.
And then that's just the ambiguity terms,
just a way of expressing it in terms of
the matrices and then policies then are just.
Softmax of negative G.
So this just basically says the
values with the lowest G are the ones
the policies with the lowest tier,
the ones that are going to have
the highest probability.
And then actions are ultimately
selected by taking the actions
that are associated with the
highest probability policy.
But we typically also control these.
The precision of action selection
with this alpha parameter here
that essentially just controls
the randomness of choice so that
the higher the value of alpha is,
the more likely the agent will
select the action with the high with
associated with the best policy,
whereas actions become less sensitive.
To the expected free energy as
alpha goes down,
so decisions become more random.
Uhm?
And another important thing to note
is that I mentioned this briefly before,
but the precision of the preference
distribution ends up effectively
waiting the tradeoff between
information seeking and reward seeking,
and the reason that is that if if
in the risk term the preference
distribution is really, really precise,
then that means that all is being equal
to KL divergent here will be bigger,
which means that policy selection
will be driven more by seeking
reward them by minimizing.
Ambiguity and that'll that'll come
back later in simulations as well.
So here, just to show this.
If she is moderately precise
then here just to show this,
here is the preference distribution.
So darker again equals higher
probability or higher reward then
if I get the null observation here
and then the choice that the agent
makes us to take the hint again
blue is the true action and black
is the action probability and
then after it gets the hint then
its beliefs updates and now it's.
Confident that the left machine is
better and so then it chooses blood
action in it and it gets the win.
Whereas if she is really precise,
so the number encoding the the
preference for a win is really big,
then instead the agent will just
take a guess at whether the left
of the remaining right machine
is the best one immediately.
So becomes really risk seeking
because the the risk term,
right the the reward seeking term and
the expected for energy dominates.
You can see it's completely
it's Gray over both of these,
so it has equal belief that
the left or the right machine.
Could be better,
but it just takes a guess anyway,
so it just doesn't go for the
hint and you can see in this case
it got unlucky so it shows the
right one but it lost.
So those on that trial
the left one was better.
The way you can see is that
retrospectively that loss.
The left one was actually the better one,
but not too late,
so just moved back to the search so so,
so that's kind of a way in which
one parameter could play a role
in helping to describe who's
more risk seeking versus who's
more information seeking.
So now if we move on to training simulations
of learning and how this can work again
in like a task that you might come in,
this sort of two arm and it asks if you just
have them have them play it multiple times,
then what can also happen is it can
just increase its prior belief that one
of the machines it's better and once
it's confident enough right away that
one of the machines is better than it
will just stop seeking information.
Uhm, so elevated confidence will reduce
this kind of information seeking drive.
And that preference precision
also modulates those dynamics.
So like first we consider like
simulations in a stable environment.
So for example,
let's say in these simulations the
left machine is always better,
and So what I'm showing here is the
agent's decision at the first time point.
So it's essentially whether it
chooses to take the hint or whether
it directly chooses left versus right.
I'm not showing what happens after
it takes the hint,
this is just whether it's odd.
Information, first or not,
and what you can see is that for
a risk averse agent,
so one with a preference distribution.
That's not very precise.
It will take the hint for several
trials before it you know gains enough
confidence that the left machine is better,
and then eventually it will start seeking.
You know,
choosing the left machine immediately,
but once it observes just a couple losses,
then it'll switch to being information
seeking again and kind of bounce back.
4th so it it changes,
it becomes information seeking
again pretty quickly,
whereas if I take a risk seeking agent
with a precise preference distribution,
it just takes the hint like one time
essentially, and then it says OK.
Now that must be better and it just
continues to select it right away.
In this case it got lucky, right?
Because the left machine actually was better,
but obviously it could have
gotten unlocked in which these
dynamics we look more complicated.
Even though it became confident equally
quickly about which machine was better
in terms of its prior beliefs at
each time point were on each trial.
Now this is an example of another
simulation that we can do with
reversal learning so we can see
that what will happen here.
As the right machine will unexpectedly
become better at a certain point and
you could see that the risk averse
agent doesn't even observe enough wins
in a row to ever start choosing the
left machine or red machine directly,
it just takes the hint every time,
but you can see here that it's believed
that the left machine were better
started to grow and then it switched
as the contingency switched and
they started to have a strong belief
that the right one was better now.
So we did reversal learning successfully.
When do the same thing with the
risk seeking agent,
it will take the hint of first couple times.
And it will choose the left machine
immediately once it becomes confident.
Once the switch happened.
Where the right machine starts to be better,
it becomes information
seeking again for a while.
Then quickly, you know,
fairly quickly comes to just start
choosing the right machine immediately,
so entering but its beliefs about
the reversal are about the same.
They're a little slower,
but so you can see that these sorts
of dynamics are really interesting,
and you'll actually see patterns
like this that might differ from one
participant to the next in the study.
So now the the final aspects of the
graphical model that that characterizes
or or describes active inference,
or this this extra kind of
bells and whistles up top,
and you don't always have to include these,
but you can and they're good to know about.
So one is this vector E and it's
just a prior policy that you
can learn it allows it allows.
Simulating habit, learning,
or habits where there's just a bias
right from the start to pick some
policies over others which builds up.
So basically the more time use,
the more times in a row you selected policy,
the more you just build up a
prior belief that you're going to
continue to select that policy.
And then again,
here is the expected precision of G.
So it's essentially like,
how much do I trust my beliefs
about expected free energy?
And you can think about it,
kind of as an inverse temperature
parameter that controls the confidence
in the agents action model.
If people know what inverse
temperature parameters are,
some people I'm sure do.
But what this effectively does is
lower gamma values and of increasing
the influence of habits so the
the less you trust your model,
essentially,
the more habits end up taking over and
you can see that in the equation here
where gamma just just multiplies G here.
So this is a lower number.
She becomes flatter and Zoe ends up
having more of an influence and that
data here is just a prior belief
about the starting value for gamma.
And so this is just saying a gamma
function where beta is a rate parameter
of a probably distribution over
gamma and then this italicized gamma
which we use is just the expected
value in the gamma distribution.
And again,
this isn't that central active
inference 'cause this isn't always used,
but gamma can be updated based on
changes in confidence after each observation.
And some people have proposed as a kind of
separate part of the neural process theory
inductive inference that these updates
in gamma or confidence and how to act
could relate to basic dopamine responses.
So so basically what you could do here
is you just include in policy inference.
Rational free energy as well as additional
evidence, essentially for whether the
policy you thought was correct before,
is still the the best one you
could think about it that way.
And then you just calculate
these updates like this.
And again, I won't go into this in detail.
I just want you to kind of know know
that it's uh, oh, hold on a second.
I'm sorry about this.
My computer is about to die one second.
Maybe it's not about time to
see if there are any questions.
And from the audience as well.
And we should also probably
stand to take a break.
And I think we had planned one
for a little while ago, so Ryan,
would it be a fine time to to to
take a break for a few minutes or?
Maybe I don't think you can hear us.
OK, I'm back. Yeah we were wondering
if this would be a good time to break.
Actually yeah, I mean,
it's a I guess it's almost too.
It's almost 2:30,
it's 228 and so yeah we can.
List I think? Yeah, no, that's that's fine.
That's fine, sorry about that.
No, you're good, you're good.
I'm glad I'm glad we didn't lose you,
yeah, so OK, now we can we can stop
here or I'm just trying to figure
out if there's any like that.
Yeah, there's probably a good stopping
point in just like a minute here,
so maybe it's just fun.
Great, yeah, let's go ahead and do that.
OK, so basically the only the
the last kind of thing was just.
Essentially you can think about you
can think about this if if you know
if you want to test this idea about
phasic dopamine responses is that
it's essentially kind of like an
expected for energy prediction error.
It's whether or not when you
make your next observation,
it's consistent with what you thought
the best policy was going to be,
or whether your beliefs about
the best policy change a lot.
So if if an observation makes you more
confident in your beliefs about G beforehand.
Then you have like a positive prediction
error that you know basically says
my model is doing really well.
My predictions are good,
whereas if F through the observations
you make really like,
strongly disagrees with the expected free
energy that basically says my policy
isn't very my model isn't doing very well,
and so you have a negative prediction
error which ends up lowering gamma
and so therefore making the model
trust habits more than it trusts
expected for energy going forward.
So you know in the slot machine.
Example, you know, for example,
if I simulate this, if they get the hand.
Then they become more confident
in the right thing to do.
Then,
and that's what it thought was
the best policy ahead of time,
then the evidence that she
was reliable is good,
you know,
goes up and you get the simulated spike.
In the basically they increase in gamma,
which again would would make kind
of like a trial by trial prediction
about about what you think about
dopamine dynamics and there is
one study by Phil Shorten back.
I mentioned briefly earlier that
tested this and did find the neural
activity in mid range opening regions
and in some of its target regions
did correlate pretty well with
these predicted dopamine spikes,
but the theory or remains to be
thoroughly tested and there's probably
a lot of things about dopaminergic.
Dynamics that you might not
be well explained by this,
so it's still very kind of
an open theory to be tested.
OK, so here's a good place to stop.
This is going to start with learning.
That's great, thanks Ryan.
I know we have a few questions that came
up and maybe we can start the next segment
by like you answering those questions
and then and then kind of going forward.
Is that sound good or happy to answer
the questions now just depends if
people wanna hang around or you know
take a take a full on break.
So I said GSP to start at 314 just to give
10 minutes to people to kind of walk around.
There is one question in the Q&A.
There's one question in the
chat to chat and yeah so.
That would be the way you guys look.
Let's let's get to the questions now.
Although while the well it's not too bad.
Yeah, go for it.
So from Mohammed drif.
Good friend of ours.
He says,
I know this is a different formulation,
but it seems that this formulation
contains with model based state
transitions and model free the habits
pieces that are reasonable intuition.
Sorry for some reason,
I didn't actually hear all of that.
It cut out a little bit. Sure no.
Sorry about that, can you?
Can you see that?
Can you see the Q&A?
I can I can click on it here.
Everyone sake it's.
OK, I know this is a promotion,
but it seems this for me like.
So it's not always treasure for habits.
Is that right?
Yeah,
absolutely you know so.
So I would say that yeah,
like EE and some science read this
like habit vector you can think
about specifically as encoding,
kind of like model free component.
It's not going to behave exactly
the way that like Model 3
versus model based RL works.
It's a little different because
I mean you become, but it does.
It does allow you to model this sort
of situation where if the agent takes
the same action over and over again.
And it keeps doing it because right it
keeps thinking it's the best policy
'cause it gets rewarded over and over
again and it gets its preferred outcomes.
Then it will just start to choose
that policy just kind of by habit.
Just 'cause it expects to
and so it will become.
They can become insensitive
to to like reversals,
just like you would get in like reversal
learning tasks where it's just hard
for a model for system to unlearn it.
And in some I mean it's in
certain cases but it gets it gets
to this kind of like strong.
Capitalization that where the gamma
term ends up kind of trading off
whether or not so the agent trust its
model and goes with that with its
explicit predictions versus whether
it just kind of falls back on habit.
That's so it's it's similar.
And then summer,
Thompson asks,
do you have any example papers
that have applied this to have a
learning or habit expressions?
Yes, there are some.
I am.
It's be probably easier for me to
just kind of send.
Then afterwards, but there are.
There are papers I could I
could point you to it for sure.
That's great we could,
I'm sure include those in the materials
that we post along with this structure.
Excellent. And, uh,
one from Jane Taylor says
maybe this will come later.
But how is metal learning
conceived of this approach?
How is how is metal learning millimeters?
And you know, I mean like,
I guess it depends.
Like you know exactly in general and
you know anything that you want to
try to model using these sorts of
approaches just involves coming up with
the right generative model structure.
To do it, you know.
So presumably you could set up the
right sort of hierarchical model
where a higher level, for instance,
has an influence on what policies
are considered at a lower level,
for example.
You know there are also situations where,
for example,
if you have a large policy space,
then you usually we usually include
something called an Occam's window parameter,
which says something very similar to
like aversive pruning were essentially,
if if a policy as you're kind of
evaluating it becomes too much
worse than like the best policy,
then you just kind of threw it
away and stop stop considering it.
So you only kind of focus on the
policies that seem the most promising
when otherwise is just too many to.
Focus on UM so,
but but in general I would
just say that you know,
if you specified some specific
metal learning problem then the
task would just be come up with the
right generative model structure
like hierarchical generative
model structure to accomplish it.
There's no kind of generic piece of
the model outside of just specifying
a model structure that would that
would do something like that,
though at least it's in the
formalism right now.
OK, and lastly,
it seems like a quick question,
again from Mohammed A.
Is there a Python version of the code?
And yeah,
so actually I mean his whole couple things.
I mean meet me in a man.
A lot of my research assistants
have been kind of in the process
of trying to convert some of this
into kind of like a probabilistic
programming language and Python
that's far from being done though,
but people at the Sussex groups, so like.
Connor Hines and Alex Johnson.
A few other people.
They they have put together.
It's still kind of data, you know.
They're still kind of adding
to it and things like that.
But there is a uh, they call it Pi.
MDP, that is a Python version,
especially just a direct Python
translation of the of the MATLAB
scripts that we provide in our in our
tutorial that can do not everything
that the that the MATLAB scripts do.
I think yet,
but but certainly a lot of it and
I'm happy to put you guys to the.
To that as well,
but just a there's just a
GitHub for it.
Well, that looks like it takes
us to the end of this segment.
But maybe we convene at 4:45
or thereabouts and and R
3:45 or thereabouts for U2,
3545 or their bats.
So 10 minutes from now and
then start again.
OK, sounds good.
It is just about that time so welcome back
everyone on for the second half here Ryan.
Is there anything we need to address
before you just jump right back into it?
A couple of the links to papers and also to
the Python implementation in the chat so.
Allison. Helen, do you think you
be able to grab those so that we
can put them on the website? Right?
Thanks man, appreciate that.
Yeah, sure, I'll go ahead and
get started then. Yep Yep, Yep,
and you can still see my screen.
Everything is fine. OK, great.
OK. So now so the next thing to to go into
here is before we move on to the kind
of practical empirical stuff is the way
that learning works in active inference.
So inactive inference and
active learning in particular,
there ends up actually being two
types of exploration when we're
dealing with the Explore exploit.
So one is state exploration right which is
seeking out information about hidden states,
which is what we've been talking about,
right? Like minimizing ambiguity or.
Maximizing the epistemic value,
so I'd just be like taking the hint to
figure out which machine is better,
but then there's also parameter exploration,
which is actually seeking out information
about parameters like trying to learn
what the likelihood function is,
or trying to learn what the trends
to correct transition beliefs are.
So, for instance,
this would be choosing the left machine.
Instead try to resolve uncertainty
about the probability of winning
with the left machine, right?
So trying to learn something about
the probability of reward given
states in the likelihood.
Uhm, in other words,
you're you're trying to choose
a machine in this case for which
you're least confident about the
outcomes that it'll produce,
and so that's driven by this added
term that gets tacked onto the
end of expected free energy when
you're including learning,
and it's based on something called a richley.
Distribution,
which acts as priors on the on the
categorical distributions we've been,
we've been working with so far,
so for people not familiar with
Bruce Lee distributions and some of
the details aren't that important,
but but what matters is that their water,
called conjugate priors for
categorical distributions,
and what that means is we can use
a Dirichlet distribution as a prior
for the cattle to the categorical
distribution that we're using,
and the likelihood function.
Make it,
I'll show you what this means in a second.
But maybe being a conjugate prior just
means that when you do inference then
the posterior will just still have the
form of additional distribution again.
So you can just carry it out.
In other words,
you can just keep learning in
the form of the distribution
doesn't change with each update.
And so the notation here,
we just say you know beliefs about
a sublease out the likelihood just
corresponds to this Dirichlet distribution
that we just know with lowercase a,
but still hold it.
It's still a matrix,
and so here the Dirichlet distributions,
we'll just have you know,
these different watercolored concentration
parameters or pseudo counts,
so they'll be like one for each probability,
right?
So the probability of losing given
the left choosing the left machine
you know probability of losing
given the right machine.
ETC and so these are often just called
counts or concentration parameters
because they just get added to each
time you make a new observation.
So for example,
say that the true likelihood,
so the true a matrix is point 2.8,
point 8.2, and we're trying to learn that.
And so the way that learning works
when it comes to learning the
likelihood is just with an equation
like the this weird essentially
adding the previous beliefs about a,
you're adding to that.
The outer product here of the
states you believe the urine and
the observation that you make,
so it essentially it's just
coincidence detection.
So for instance,
if I believed that I was in the left
machine state and I observed a loss,
then I just add account to a one here.
This gets bigger because if I
observed a coincidence between
left and loss then that should
increase the probability that left
will generate loss in the future.
So it's very similar to kind of
heavy and learning in this kind
of coincidence detection sense.
So there's like a practical example.
Let's say that my concentration parameters
start out a really low flat value,
so just point twos right along the way.
So I know nothing.
If I then observe a win or I'm sorry,
I observe a loss,
then what will happen is I literally just.
If I believe I was in the right
state and I get this observation
letters out of one right?
So now this is 1.2 and the
rest of these just a .2.
And now when I softmax over these
then this will end up having a
higher probability than this.
I'll believe the right machine is more
likely to generate a loss and that's it.
Come and then you can add things
like parameters to this that you
can fit for participants, right?
So this is a this Omega here as
we call it forgetting, right?
So it's just a value between zero and one
and lower values indicate more forgetting.
Essentially,
it just makes it so the counts lower
that number is the counts kind of
stay lower and lower and move back
towards the prior values and then
you can also add a learning rate
which basically just scales accounts right?
So instead of adding a one it might
only have like a .7 or something
like that depending on.
Learning or so just slow,
is how quickly beliefs change.
So again,
just a number between zero and one.
And so they expanded for energy expended,
expected for energy.
I mentioned ends up just including this
novelty term that takes into account what
those concentration parameter values are.
And so here's what the new expected
for energy equation looks like.
And again,
you know,
sometimes you look at these things and just
like so cryptic like what does this mean?
But actually all I did
is just add this right.
So this is just the risk term
for before this is just the
ambiguity term before and this.
Although it looks complicated
at the end of the day,
it's literally just the difference
between the observed the difference
in a after I get an observation
and I'm going
to state versus my beliefs about
8 before I got that observation,
I believed I was in that state.
In other words, the bigger this number is,
the more I expect the distributions
in A to change, and the bigger this
number is because the subtracted,
the smaller G will be.
So in other words,
I'll be driven to move to states.
That have the lowest concentration
parameter values essentially,
and then when you translate
it into matrix form,
all you're doing is adding
this as times W thing here.
And that just matters because
W equals this and and what
matters here is just these sums.
So essentially what this is is it's
just some summing the values in
the columns of A and then making
it so the larger values in a are
the ones that contribute less.
So so again, just kind of intuitive example.
Let's say that I have a matrix like this.
Where concentration parameters
are 50 and 50 and .25 point 25.
So essentially I've observed the
outcomes of going to the left way more times,
but notice that both of these
distributions are flat, right?
So they're equally ambiguous.
So the epic Unity term is going
to be the same for both of these.
The novelty term,
because these counts are really big
and these counts are really small.
This term is going to be quite a
bit bigger for policies that move
you to the to the right state,
because you've just just have way less
confidence in what that distribution.
Really is, even though your beliefs
about both of them right now,
or that it's flat,
so that's really it.
You're just driven to select
to move to states that have low
concentration parameter values,
so you can learn more about
what the probabilities are.
So just to give an example of how the
dynamics of this end up looking and we
can take this is our previous study.
I'll show you guys in a bit
from a three armed bandit task,
which is a little bit different
where the just doesn't know
which which band it is best,
and then it just kind of has to trial
and error figure out which one is best
and and in this case when we include
the novelty term you can see that the
agent is actually very strategic, right?
So the first time it's beliefs about
which bandit is the best one, right?
Just grey across,
there's no idea.
It just chooses 1 randomly,
but then it wins.
It gets a green here and so now it's very
confident and it just chooses that again,
but now it loses.
But now it doesn't choose randomly.
Right now it says OK,
I've gotta pick one of these two
'cause those are the ones that haven't
seen yet and then it gets a win.
And then it says that that one,
but then it gets a loss and so it is OK.
I'm going to check the
one I haven't looked at
it all yet. It loses and then it jumps back.
Here it gets a win and then just ends
up settling on choosing the third one.
Which is the best you can see.
This is very strategic
information seeking if instead.
I'm a kind of like a room work,
only model, right?
So let's just say I get rid of those two
terms that I just have the risk term,
and it looks like this instead, right?
The agent has no idea pics.
One loses, no idea.
Pics one again. No idea.
You know loses 61 again, etc.
So there's no,
there's no strategic information seeking,
and so it doesn't do as well.
So that's that's what the
novelty term ends up doing.
So those are the kind of summary
part now for kind of part one,
which obviously I was
hoping to finish before.
Is that, you know,
active inference describes perception,
learning, and decision making.
In terms of this approximate Bayesian
inference using expected for energy
minimization and message passing,
and it does that in a way that's
biologically plausible and then
and therefore it can generate
neuroscientific predictions and it
includes this reward seeking as well
as this information seeking drives
can be helpful in certain past
contracts that involve uncertainty
and it includes active learning
which we just went went over as well.
OK, so now onto.
Empirical applications,
you know.
We're hopeful for stuff again
gets practical now.
The idea here as I kind of briefly
mentioned a couple of times,
is that each of these models have a
number of fixed parameters that can
account for individual differences, right?
If I change a learning rate,
behavior will be different
in a particular way.
If I change action precision,
it will also change in a particular way.
And So what we want to do, you know well?
I mean, here's some examples like preference,
precision, precision,
special state priors, precision accident,
precision learning rates.
Forget for 8th grade,
there's a bunch of these things
right that you could try to
identify the values of those was
for each individual participant,
just by fitting a model to
their behavior on a task,
and then we can use those parameter
values as individual difference measures
like a between subjects analysis.
I was like example.
Listen to action perception,
which remember is just this
thing that controls how random
actions are given the best policy.
And higher alpha here means that action
selection is more deterministic, right?
It's more value sensitive.
So say we pick a value of alpha equals .1
and then so here this is the probability
of choosing the right slot machine.
And let's just say that this is the
expected free energy of the left
machine minus the right machine, right?
So the more it moves over here,
the lower the free energy is of
the right machine and therefore
the higher the value is.
So at .1 you can see that the
predictions of the model really.
Mark all that good compared to the
actual date of the participants.
Choices health equals 5 Right does better,
but the best fit.
If you just kind of wiggle it
around until you find the best
fit is healthy equals .3, right?
So just by finding the value of
that parameter that best accounts
for a reproduces the participants
data in the model,
that's what we're trying to get,
so that's the value.
At least three would be the value
that describes that participant best,
and the idea is you can do this for any
parameter in a generative model that
you build for a particular behavioral task.
So you want to build one or more
of these generative models and do
some kind of model comparison.
There's no way to figure
out which model fits best.
You want identify the parameters
that are of interest to you.
You know fit the models to
participant behavior by moving these
wiggling these parameter values
around until you find the best jet,
and then once you find the best fit,
then you would use these parameters
for between subjects analysis and you
know in the past we you know we've
been 'cause my lab is kind of focused
specifically on trying to apply
these parts of models to empirical tasks.
You know,
we've we've found them helpful in addressing,
you know, some some.
Interesting questions,
so in one study, for example,
we we looked at substance
users using an explorer.
Quite task actually,
that that three armed bandit ask.
I mentioned briefly and what we found is so
this is what the task actually looks like.
You know,
you could just keep choosing option one,
two or three and either get
a green ball which is a win,
which means you win some money or red one,
which is a loss and you just kind of
trial and error until you settle on one.
And there's a particular generative model
structure that we designed for the task.
And what we found at the group level
was that action precision was higher in
healthy controls and substance users
and learning rates for losses were
higher in healthy spent substance users.
So so in other words,
in substance users,
when they got a negative outcome,
they were they learned more slowly from that.
So you know one thing,
we were kind of excited about is,
you know, that could help address this
idea of why substance users continue
to use substances despite negative,
despite the negative life.
Consequences?
That they are that they lead you.
So if you learn more slowly when a negative
outcome happens and you also with action
precision just behave more randomly.
What a positive outcome happens then?
I mean, that's kind of a recipe for
continuing to do maladaptive things.
And another thing we're
really excited about this.
This actually is just a
preprint we just put off,
but we followed these people and
looked at them again a year later.
And we looked at whether the baseline
parameter estimates that we got could
predict predict anything about symptom,
severity or substance use
severity and a year later,
and what we found was
actually in stimulant users.
An opioid users.
If you took the baseline learning
right from losses and then you
could predict symptom severity a
year later and same thing with this
baseline insensitivity to information
which essentially like how much
information seeking Dr they have.
And then you know,
and then you can just show
right in this task.
An example of model fitting, right?
So if I hold all parameters constant,
but I give a preference mission of
.5 and these blue dots are are an
actual participants behavior choices,
you can see that this doesn't
fit well at all right?
So darker,
equal style probability is not
matching participants behavior at all.
Whereas if I put a preference precision
of 1.92 then all of a sudden the model
predicts with really high probability.
Most of the participants.
Action is pretty well,
so that's the kind of thing
we're looking to do.
And what we did in that study.
Another study that where we've applied
this empirically or a couple studies
is this task generating a model of
this task for approach avoidance,
conflict behavior.
And we did this in depression and
anxiety and substance use disorders.
And in this task and it's kind of fun.
Like I mean, you just kind of move,
choose to wear to move
this avatar on this runway.
Like if it's closer to like a rain cloud,
then that means you're going to see
a really negative image and sound.
And the sun means you're going
to see a positive,
relaxing image and then the
red corresponds to how many?
Once you get Daddy is basically how how
much are people willing to approach this?
You know the side where they're going
to see a really negative thing,
but win points, right?
So that's just it's just planning.
Based on this approach,
avoidance conflict and what we found
is that using a model you could
actually dis entangle two different
parameters that contribute to conflict.
Or you know the conflict under
you know they with this decision
because it's focused essentially
how how close you move to one
side or the other makes it hard,
just higher probability that you'll
get one outcome or the other.
And So what we found was that.
And this is at baseline and also a year.
We found that healthy controls
showed higher emotion conflict.
Actually, both at baseline any year later.
Then depression, anxiety,
and substance users.
So substance users and depression
anxiety actually somewhat kind of like.
Flat or you know they just didn't care
as much in some sense about avoiding it,
but they also showed significantly
higher levels of decision uncertainty,
so the prior owner,
the beta prior on gamma.
So there were just a lot less consistent.
It was like they didn't really
know as confidently what to do,
but what they did do tend to avoid less,
and so that was again like consistent
with pretty high intraclass
correlations here later.
And so, and you could think about this.
Conflict is also just
like reduced preference.
Precision is essentially
what it was in the model,
but it was the basically how much
you avoided versus one of the points.
And the next thing that we've
done is a very different kind of
application is looking at perceptual
inference in interoception.
So for this what we did is we just
applied this heartbeat tapping task
where basically everybody just be asked
to pay attention to their heartbeat,
and then they just tap a button every
time they think they feel their heartbeat.
And for this we could use a
assembler like Hidden Markov models.
Basically,
essentially what I showed you only
there's no explicit policy selection,
you just assume the agent picked
the thing with the highest posterior
probability and perception.
And what we found here that we were
you excited about is that there was
actually no difference in prior
beliefs with respect to heartbreak
perception, but the people with anxiety,
depression, comorbid depression,
anxiety, eating disorders,
and substance use disorders across
all of them.
They showed this inflexibility in
updating sensory precision from
heartbeat perception.
So essentially,
when we told them they could guess
or just they shouldn't guess,
everybody's values were similar,
they only had them.
Hold their breath,
which makes the afferents or heartbeat
signal coming up from the body makes
it kind of stronger, more precise,
the the estimates in the model,
right?
Peoples beliefs like it updated so
they they they had higher introspective
precision estimates and healthy.
But it just Inflexibly stayed low
in the other group, so it was.
So it's like trans.
Diagnostically there was kind of a rigid,
inflexible beliefs about the the reliability,
essentially of the signal coming
up from the heart. Dumb and dumb.
And then the last one that I'll
tell you about is one that we just
very recently published that
actually started to test something
about brain responses,
and it was almost an identical models
of the heartbeat tapping model,
except that in this case it was looking
at gastrointestinal interoception,
and so we did is.
And this is Skype call,
so you know who was the one
who came up with this idea and
figured out that we could do this
so you know all credit to him.
He found this company that makes
these capsules that are they have a
little motor inside of them that that
essentially vibrates in a pattern
that you can like pre programmed.
And they're used clinically to
help with them with Constipation.
But what we could do is we could just
specify a specific timing pattern,
and so they could feel vibrations
in their stomach for about 1/2 an
hour period with a particular.
A sort of timing that we sat and they
would just push a button every time
they felt a vibration and there's like.
And so in that model, and when we fit it.
But when we fit behavior to the model,
there was five parameters in the model.
There was a couple different learning
rates based on whether priors going to
decayed more quickly or grew more quickly.
And also there was a sensory
precision parameter like the
precision of the likelihood and
also a precision of prior beliefs.
And what we found was was that interoceptive
precision was positively correlated
with CRPS in posterior brain regions.
So essentially the more you
trusted the signal, the stronger.
Belief update was in the brain based on
posterior peas and learning rates modulated
frontal CRPS and prior beliefs stronger,
probably sexually suppressed.
Friendly arpis,
all of which was pretty consistent with them.
With you know,
theory about hierarchical inference.
So, so again,
we're you know excited about that.
And now you know these are examples
of just fitting models to behavior
as kind of trade measures.
Another practical empirical thing
that you can do is you can use the.
The neural process theory
you know where this?
This just describes the message passing,
and this is just updating beliefs
about States and you know again,
we we've actually gone over each of
these now and and you can use these
to make trial by trial predictions.
You can get predicted neural time
series essentially and I could show
you like what this might look like.
For instance when we go back to our
simulations about the risk averse
agent and the risk seeking agent,
you can generate predicted
ERP patterns and you can also.
Generate this predicted expected
for energy precision updates.
So the gamma you know like the
thing that there's been proposed
really to dopamine responses,
whereas in this case you predict a
different pattern, a different phone.
Yeah,
time series of of ERP S and precision and
action or policy precision updates are
expected for energy precision updates.
I'm saying is you could generate
these regressors and feed them
in right as a you know regressors
and say like in fMRI analysis.
Is this another example from
a simulation study?
We did have a concept learning
task where you generate this
kind of interesting increase,
predicted increased amplitudes in ERP's,
and a much more kind of fluctuating
complex pattern with respect
to the precision update.
The expected for energy precision
updates at gamma.
Another another example that will
just allow me to show you the
way that you can use just pure
perceptual tasks with them.
With this model is these global
local global paradigms,
or you know,
like auditory mismatch paradigms
where at the first level all you do
is you just play like a few tones
in a row and you could have the 4th
tone be expected to be the same one.
Or you could play a few times
in a row and have the last one
being unexpected tone.
That would be like a local
total local deviation.
Local standard, but then at level
two you can have sequences, right?
So for instance you could have
a case where you have three
low 3 high tones in a low tone,
and if you do that several times in a
row then the agent will start to expect
to hear the low tone on the 4th one.
You do that, then eventually when
you play the tone where it's
just saying it's high each time,
then that's unexpected,
so you can have this case where,
for instance, at the local level
there should be no prediction error.
Because it's just the standard,
but at the global level,
it's the unexpected sequence,
and so these models.
You know this sort of tasks can be captured,
even though it's a purely perceptual task,
right? There's not really
interesting behavior involved.
People just kind of stay
right what they hear.
But what you can do is set
up a hierarchical model,
which is really, you know,
it's nothing all that fancy.
All it is is instead of taking instead
of just using observations at the,
you know down here,
you just replace those with states
at a lower level in the model.
And what that means is business.
You have these second level States
and these personal States and the
first level states just act as the
observations for the second level.
And what that also means is that the
second level operates over a slower
time scale than the first level.
So for instance,
each state at the second level puts a
prior on a full trial at the first level.
So essentially each state at level
2 is a full trial at level 1.
And so this operates.
So the beliefs about States and the
vertical neural responses at level
two are much lower than at level 1.
For the long run trials,
for each second level trial,
and then you can set up.
You know this basically the
same neural implementation,
just kind of stack another one above it.
There's no additional sort
of thing you need to do.
So in this case,
the higher level would represent
beliefs about sequences,
whereas the lower level represents
beliefs about individual tones,
and so you can generate.
You can use these to generate predictions
where the lower level corresponds to short,
like sort of fast.
You know,
like early ERP S whereas the second level
it makes predictions about later ERP's right?
So these could be like you know any
one hundreds and things like that.
Whereas this later this horrible thing
to be more something like a P300.
And so in this case you can see at
the local level you have high tone hyphen,
hyphen, low tone,
and you get this increased.
You know stronger predictive ERP,
whereas up with the second
level the thing starts out.
Once it gets a high tone first one,
it's equally likely,
and it's mind that a sequence
where it's all highs could happen,
or where it's three highs in awhile.
But when it gets to low one then all
of a sudden it updates its beliefs
that it must be the high low sequence,
and then you get this predicted ERP.
Waveform because of the jump in belief here.
You do that over and over again to have
after learning to expect the deviation,
say 10 repetitions.
Then you just do all the high tones in a row.
There's no predicted ERP earlier P at the
first level, but at the second level,
right it initially had the prior belief
that it was going to be high low, right?
Because that's what it's heard
the most times.
But then when it gets the 4th high,
all of a sudden you have this big switch.
It's a much faster larger change in belief,
and you predict a stronger late
period late here, feel like.
300 and if you just send them together
and clean them up a little bit,
then this ends up perfectly reproducing
why you expect from what's been observed in
previous studies in terms of like mismatch,
negativities and P3 hundreds depending
on whether you're looking at the first
level or the second level waveforms.
So you can also do that as you can
see this kind of like trial by trial
level so you know those are all
just a bunch of examples.
You know those hoping to give away that
these models can be used in practice.
So now I just wanted to provide.
Some actual steps for you know
how how we have done it.
You know one way that you can do it,
you know.
So the first thing is you you want to
select a particular parameter estimation
algorithm right out whatever way you
want to use to find the parameters that
best reproduce a person's behavior,
and then you want to perform
Bayesian model comparison.
You know out of all the different
models that you try to find which
of the models fits best.
And then there's a bunch of model checking
things you know that you should do.
So,
for example,
we generate simulated behavior
using whatever the winning model is,
and then you can just estimate
parameters again on that simulated
behavior and see whether the the actual
parameters that generated the behavior
are similar to the one that's ones
that the at that you know estimates.
And.
And so you can just look at
how strongly there correlated,
for example,
and then and then you can use the parameter
values for between subjects level analysis.
So there's lots of different approaches
to doing parameter estimation.
The simplest kind of thing to
do is just a grid search,
right?
So you just look at a bunch
of different values for say 1
parameter and a bunch of different
values of the number parameter.
You just find the one where the
probability of the data given the model.
Is highest.
But that only works in in
fairly simple models that
have a small number of parameters.
So when it's more complex when
when you have a more complex model,
then there are other kind of more
sophisticated approaches and the one
that all focus on here just just as
one example that we've tended to use
is an approach called Variational
Bayes and the the reason that this is
somewhat interesting part of part of
why we kind of like using it is also.
Like I mentioned, it works.
It works well for models with order
number parameters, but essentially.
But it does the exact same thing as
what that's how perception works
in in active inference, right?
So it simply uses variational for
energy minimization as a way to find
the parameters that best explain
of participants behavior.
So it's the same thing that
active insurance policy is.
And so you just start by specifying
some initial estimation priors.
So prior values for what you think are
the most likely parameter Browder values,
and then and that will give you
and you specify mean at a variance
and then estimation just involves
moving parameter values around from
prior to posterior over parameters
until you find the one that has the
lowest variational free energy.
And just like in active inference models,
this ends up maximizing accuracy
while also minimizing movement
from prior values complexity,
so this ends up.
Deterring overfitting by minimizing
for energy in the same way as that happened,
that perception in active inference models,
and so you could think about this as
just trying out some prior some prior
you know combination of parameters
you know trying another value,
finding that one has lower energy,
trying values around it.
Great for another set of values and
eventually you find some some free
energy minimum when you find it very.
That's the combination that we say
best explains compilation parameters
that best explains the participants
behavior while not while not moving
priors farther than you need to
to prevent overfitting.
So you know,
in the in the tutorial scripts
that we've provided,
you know we kind of explain this
in more detail,
but there are things you want to look for.
So like I'll run this in a script
if I have time.
But you know,
this is kind of as it's fitting.
You'll see that the these are negative
for energies of spike going up.
The way it's going up,
but it'll find free energy minimum,
and if it kind of just gradually
goes up in Plato's like this,
then that tells you the estimation
is probably going pretty well.
Whereas if it kind of goes up and
down and fluctuates wildly and has
trouble converging that you know,
that might suggest something's going wrong,
like you might have started with that.
Bad priors for the parameters or there
might be something wrong with your model.
And so.
Just to go a little bit more into the
you know the Bayesian model comparison.
Yeah, the idea is typically what we want to
do is create several different task models,
then compare them to find the best
model and then use the parameter
estimates in the best model as
individual difference measures.
Now, in active inference models,
often the models you can compare
can have the same model structure,
but maybe just include or remove
different parameters you know,
like just include only one learning rate,
or see if two separate learning
rates work better.
You know for example and compare
those and then just compare the
variational free energies of each
fitted model for each participant.
And then the model with the lowest frenergy
across participants is one of the best.
Explains the data and this is
just an example from the three
armed bandit study I mentioned.
We tried out ten different models
and the model that one in Bayesian
model comparison was the one
that had the actual precision,
their reward sensitivity,
the institute and sensitivity information,
and the and the separate learning
rates for wins and losses.
And so you know,
as I said,
once you have a winning model,
I mean it's important to make sure
that the parameter estimates in your
winning model reliable and so you can.
Like I mentioned,
generate simulated data using different
parameters and then try to estimate
the parameters from simulated data and
see if they match or are close to in
some way with the what the simulated what,
what the what parameters were that you
actually used to generate the behavior.
And so if the estimated parameters
are similar to the true parameters
you used then then you know you're
probably in good shape and they're reliable.
If they're not similar than you know,
that can suggest that they're not reliable.
So,
for example,
it might mean that there are multiple
parameter combinations that could
explain behavior equally well.
For instance,
maybe a higher learning rate and low
action precision explain behavior
just as good as low learning
rate and a high action precision,
and if that's the case,
then that's problematic because you
know you there's not one unique set of
parameter values that the best explain
participants behavior that you could use.
And you know,
to give to give a sense of this hypocrite,
search for possible all you know,
say it,
say that the grid search
revealed probabilities of
data given a model like this,
where you know that combination and
that combination is equally good.
That's the kind of thing.
Isn't the case or you want
to make sure you address?
And so here you know some other outputs
that our model scripts give is.
You can do things like this,
so you can say this is the
prior value for alpha here.
For action precision the true value was down.
Here that generated behavior and you
can see the posterior moved toward that,
or did pretty well.
So that's pretty good, right?
It moved in the right direction.
Same thing that the true value
of the generated the pressure.
This is a reward seeking
risk taking parameter.
It's a preference precision that the true
value was up here in the prior was great,
so it also.
Moved up in the right direction,
so let's suggest it's doing pretty good.
Another thing you want to check,
and this is another thing
that it spits out is.
They covariance matrix,
so this basically says on the off
diagonal to how correlated were the
changes in alpha and the changes in
risk seeking or preference precision.
And you want to you want to you know
it's it's you want to make sure that
these aren't too highly correlated,
so these are actually independent
aspects of explanation for behavior.
This is another thing that
our model scripts spit out.
I mean this is just showing what are what,
our model. What are scripts will do?
Is they generate.
They pick a few different combinations.
Grammar values I'm just showing one
example of different values of alpha
and then it generates patterns of
behavior and the two armed bandit tasks
under those different alpha values.
And then we use variational Bayes
to estimate those parameters and
then you just want to see if they're
correlated so you can see in this way,
in this case they estimated alphas
are very close to the true alphas.
The correlations .94,
which suggests that that's pretty reliable.
And then at that point you can just simply,
you know,
take the posterior mean estimates if
you want and use them as predictor
or outcome variables in some second
level analysis you could say,
for example,
is learning rate faster in those
with higher anxiety or as depression
associated with more information
seeking things like that but?
It is important to note is that doesn't
just give you the posterior means,
it also gives. Access.
Ultimate surrender, we estimates,
and it could be useful and some
approaches to incorporate that.
Well, I thought. You can do it that way.
Estimates let's hear your variance.
In other words,
if there was less confidence
in those estimates,
and so there's some example of
a way to do that with you is.
Approach is called parametric Empirical
Bayes that uses Bayesian general.
Your general Ledger model that
incorporates both the means of
the variances for the posteriors
and so are our scripts will also
run examples of PEP or you for
between subjects analysis.
And basically what it does is
it will compare models.
Double compare models with different
numbers of predictors predicting
your parameter values and and then
it will just compare models and only
keep the models that that have.
It'll keep the reduced model that
has the best evidence that will
get rid of parameters in a model
that that are making the evidence
worse or the full model could win.
If the full model is the best,
provides the best explanation for
the between subjects data and so
that's the last thing that we
show here in this figure in the
paper is it will just show you,
for instance, the full model here
and then the reduced model here.
And so this is estimating when
it's estimating 3 parameters.
It's taking this example it's
measuring or it's estimating action,
precision,
reward seeking or preference
precision and a single learning rate.
And so it's just showing that there's only.
In the kind of fake data that we
generated with different learning rates
and different reward or risk seeking
or preference precision parameters,
it did in the the winning model.
It did show lower.
It did show a group difference in
preference precision and then we
also just made up fake age data and
somehow it also identified that age
contributed to a couple of these parameters.
I guess this is just fake data that is
just used for an example in like 6 people.
Six simulator participants.
So just just to kind of,
you know,
like reiterate them steps for this approach.
This example approach that is
walking you through is you'd select
some estimation priors with mating
periods is based on participant
data for each model under control
duration sector of updating the
gradient descent shows a smooth
convergence with free energy minimum,
then perform Bayesian model comparison.
To consider this model and then
generate simulated behavior in the
best model using the parameter values
that you found in your participants.
And see if they would see if the
estimated parameters correlate
well with them and and then confirm
that your posterior estimates
approached the parameter values.
That Umm,
setting for the simulations and
confirmed that the posterior
covariance is aren't too high.
Like I said,
those are all just things like
that in the figure,
and confirming that the generative and
that's made friends for life so that
there's all this stuff in the figure today.
Plan spoil.
And then you could use pad as one
example to do run regression for
group comparison analysis that
they can do account for Santa.
During estimation, so now I'm not sure
exactly how much time we have left,
but I was hoping to get a sense
to walk you through some of the
some of the actual code so that
you would know get more of a sense
of you know how to how to use it,
and how to build a model.
How much, how much time?
How much time do we have about 30 minutes?
Better about 30 minutes.
So OK great and and OK so I'm gonna need to.
I'm gonna need two well before I do this,
I just want it because I'm
just leaving the PowerPoint.
I just want to say thanks and I want to.
Acknowledge my funding sources,
and again acknowledge Carlin and Christopher.
So let's see if I can, so I might need to
stop caring and repair my whole screen.
Let's see if I can do that.
So, so I'm in MATLAB now
and I already have open.
The already have open the
the main script and and so.
So all of these scripts that this
man script and the supporting
scripts are just in my GitHub
that I we provided the link for
and the information and I'm also
happy to to point anyone too,
but the way the way that it works is
that it's set up so really easily you
can just do pick simulation 1234 or
five and if you pick simulation one.
Then all it will do is it will
just generate single trial.
It'll generate single trial data for
the for the two armed bandit you know,
plus the hint task that I went through.
So if I click play on this just
as is just the same equals to
one then you know take a second.
And then you know we have included
these plotting scripts, so where this?
I mean, it's literally just the same kind
of stuff that I was showing you, right?
So this is just saying at the end of the
trial the agent thought was confident
that the left machine was better.
You know, it chose to take the hint and then
chose to go left to choose the left machine.
You know these are the
different outcomes, right?
So whether it observed the hint or not,
whether it won or lost,
there is also technically this
third album modality that switched
kind of observing its own.
Voices, which is just needed for
kind of technical reasons and
you also have the you know,
they simulated expected free energy
precision updates and the way that
posterior beliefs over policies change.
So here the second one would just
say that starting at time .2 the
thing had very precise posterior
beliefs that policy two was best,
which was taking the hint and
then the sequence of taking them
to then choosing left.
So that's that's what that is.
The other one is the neural
process theory figure,
which does the same thing.
It will generate.
You know these sorts of plots,
which I showed you as well as
these sorts of thoughts and the
simulated local field potentials,
or or ERP's,
and then just another part
of the dopamine responses.
And this you can kind of ignore.
We explain it briefly in the TOTORIAL,
but it's not very.
It's not very relevant here.
Then so it will do that and and you can play
around with the different parameter values.
So here this is the risk seeking parameter,
so the preference precision you know.
So I made this not that high,
right?
But it's a I say I make.
At the same time again.
And while she is exactly the sort of
effect that I showed where now right,
the thing just guesses,
it just gets a great way sometimes is
really rich speaking because it just,
you know it has to get the big reward now.
And here you can see the simulated
firing rates and things that again
match just the exact same format I
showed in the in the in the talk.
Uhm, and if you run a simulation 2.
If you run simulation 2.
It will smell like files,
UM weather app contacts
with those belong to test.
And so here.
It's gonna take a second 'cause it's running.
I think 30 trials or something like that,
and this will just again it will reproduce
all the sorts of things that I showed you.
So the behavior given the particular set
of parameter values, wins and losses.
It also shows the free energies
after each observation.
The predicted time courses,
and then the way that beliefs
about context change overtime.
Uhm 3 will do.
The three will do their reversal
learning one.
Which I am. Again, I was really hoping
that just going through the talk right
now it would be it be clear and easy
to you know to interpret these plots.
But this is just a reversal,
learning version and and again you can
just kind of play around right with
different parameter values both up here
and just down in the actual model,
which I'll show you in a second
and just see how behavior.
See how behavior changes.
And number #4 here will actually
run parameter estimation for you.
So will simulate data for a participant
and then it will run variational Bayes and
estimate the parameters for that person.
And so I mean I could show you
kind of a little bit what that
looks like in relation to.
Yeah.
What I showed in the paper and the
figures I showed in the paper and
you can see what it will do is.
It will most.
So the the evidence for the model
probably of the data under the model
under slightly tweaked parameter values,
starting with some prior values
that I set up, and then after it
finishes one round of it's kind
of like just evaluating locally,
but the differences and evidence are.
And it will just kind of do this
over and over again until it
converges and what you'll get is
something so it will spit this out,
which again is just more or less the same
as what we show in the in the tutorial.
Here you know, I wish this wasn't
going so slow so I could show you,
but but ultimately what it will do is
we will just update after each iteration
of variational free energy minimization.
And it will give you something that
that looks like this right where it
will smoothly converge over time and it
will tell you what the deviations are
from the prior to the posterior score
for each parameter that you include,
which is just showing there.
And and yeah, so Oh yeah, now right.
So now it's evaluated the expected free
energy relative for the first time and it
will just kind of keep going up like that.
So I can just let this go for a second.
But and then if I were to run simulation
5 it will just do the same thing but it
will generate simulated behavior from six
different participants with six different
combinations of parameter values and so then.
It will allow you to do model comparison
between models that have been assumed
different numbers of parameters.
It will show you how to run correlations
between generative generated an
estimated parameters and it will.
It will do.
You know, essentially everything
that I that I walked through,
and obviously that takes a little bit of
time because it has to do this six times.
But once it does then it will spit out and
it'll spit out some so I can just stop that.
Now, since you've seen that it will spit
out a couple data coupled data files that
you can then use to do certain things.
So for example down here and I'll
walk through this more in a second,
but I just want to show you.
It will save, it will save where are you?
Yeah it will save files.
That's a two parameter model estimates
versus A3 parameter model estimate,
so these are models that include versus.
Don't include learning right?
And it will also include save these
files called GM two and GM three.
So the these are the. C M Switch.
Again, I'll explain in a second
for the two parameter model and
the three parameter model after
it was estimated for each person
and so then what you can do is.
This will allow you you won't have to rerun
this every time to get these these values,
but once you have them.
Then I can just I can just load a little,
do it for me I guess.
I can just load.
Let's say I do the two parameter model.
So I can I can load that in and
then if I want to then I can
just take those values and I can.
I can show you. And I can spit out.
I can spit out the.
You know the sorts of
correlations I showed, right?
So that's like the true
generative parameter for alpha,
and these are the estimated alphas,
but I can do that for others too, right?
So this would be for risk seeking
in the two parameter model.
And you can see here also,
the correlation is quite high, so .95.
And you know, I can also do
that for a 3 parameter model
that included learning rate.
Oh something, oh I didn't load the
three parameter one, that's why.
I could do the same thing,
but point is when you actually
run when you actually run it,
so you can see a learning rate.
Actually, the estimate,
like the recoverability
wasn't quite as good, right?
The correlation was only .58 which basically
says is that you know these two values.
The true values would .2 and
point for warning, right?
How, how estimable they are dependent on the
values for the other parameter settings.
In some sense you know this isn't.
Yeah, terrible and like a
large number of subjects.
You know if there's more
than just six people,
presumably this would be a
very significant correlation.
But sat here is impressive,
as though the other two.
So and then the other thing that these GCMS,
well that will spit out is they
will allow you to do pep and so
here you can just choose to make
GM to choose the two parameter
GM or the three ground or GM,
and these just spit.
These are just structures
that get fed into Pebble.
And once you and then so down here,
once you have also here I'll show
you like once. Virus goes away. Uhm?
So if you select. You can turn
pevan right from the start, right?
If you make have one here and
then you run simulation 5 then it
will do pet for you automatically,
but once it spits out these GM two
or G&G M3 for the different models,
you can just run this section separately.
And it will and it will do it will estimate.
Do the pet estimates for you and just
so people know some of these are.
Some of these outputs are because these,
these these scripts draw on the.
On the dynamic causal modeling
scripts and SPM.
So not all of the outputs are really
all that useful for this application.
You know the ones that are useful.
Or for instance,
the ones that show the full model and
the reduced model for different for
the different predictors in the model.
But probably the most useful thing it
spits out is this review parameter
inspector thing where where you can just
select whatever effect you're interested in.
So let's say I'm interested
in effective group, right?
We made up two groups that had
different risk seeking levels,
I think.
And I can do that and you can see that
these are just the two parameters and
it doesn't show you what they are
because it just codes them as numbers.
But this is alpha and this
is risk seeking right?
'cause this is a two parameter model
and this is just showing there is a
group difference in the posterior
estimates for risk seeking between
the two groups that I included.
Whereas you know if I just look at
the effective age then apparently
there's some effective age.
But well, actually I should, I should.
Show you this,
that and there's no thresholding
going on here,
so this is just showing every time,
whereas I can set a post your
probability threshold, you know.
So if I for instance say I'm only going
to say that something is significant,
if the probability is greater than .75,
then all of a sudden that goes away,
whereas the effect of group you know,
even if I make this thing probability is .99.
So then it goes away if I say .95 then
it's still there and that's based on the.
For energy and you can also just
make it based on the probability.
Just a normal cluster probability as well,
in which case let's see if it's still.
In this case the posterior
probability being greater than .99.
Still,
I'd still survives so you know you
have to just decide what sorts of
significant structurals in terms of
posterior probabilities you want
to accept when you're doing them.
When you do these sorts of between
subjects analysis to look at differences
between groups and parameters,
or correlations with things like age or sex.
For anxiety level or things like that.
But again, this is.
This is just one approach,
you know,
there's there's lots of other ways that you
can. You can use these.
These sorts of these sorts
of models for these sorts of
parameters at the group level,
so I'm just kind of illustrating
one thing that we provided that
is useful in that it it allows you
to take advantage of the both the
posterior mean and the posterior
variance of the of the parameter
estimates that you get for each
participant for each parameter.
So that's kind of broadly the
sort of functionality.
Of of the main,
the main tutorial script that we provided,
I should mention there's lots of
other little smaller tutorial
scripts in there as well that
just link to the tutorial paper,
but but this one is just the kind of
the main one that goes through most
of all we walk through like we also
have another script that shows you
how to build a hierarchical model of
the of that auditory mismatch task,
for example,
and we have other ones that
allow you to like,
calculate and play around with calculating
prediction errors and things like that.
But now you know,
since we still have some time.
Yeah,
you know the main point of these
scripts or the most important thing to do,
you know?
So we we obviously we provide you the
scripts to do parameter estimation
using variational Bayes and things
like that for practical purposes and
later in the script down here, right?
We kind of show you how to do that,
right?
So there's you know down here,
for instance, you can,
you can just pick right which parameters
you want to estimate and things like that.
But but the the main sort of first
thing that you know is important.
Most important for learning to
apply these sorts of approaches
is learning to build a generative
model for the specific task that
you want to you want to use or
that you have data for.
And so this is just an example,
sort of a detailed,
highly commented example that
describes in detail exactly what we
did to build the generative model for
that for that slot machine the hint slot,
machine task and so I can
just walk through it.
You know, since we have the time, you know.
So the first thing we did is we just
specified how many time points you know.
So how many cows?
And so we said that equals three,
'cause there's three time points,
right?
So there's the maybe I'll just
run one so I can walk through
this at the same time.
Brian, just letting you know that
there are some questions in chat
if you know there are questions.
OK, so so I yeah for for this.
I mean it might be.
Yeah it might be fine to just take
take questions as we as we go.
So yeah I mean feel free to uh,
let's see what?
To open the chat I could also
map iteration Riddle out as well.
It's a pretty simple one,
so this is also from Mohammed,
and the question is regarding
the behavioral data fitting.
Can you use things like reaction times
for example to fit different models
as opposed to just choice the choice?
Yeah so. Yes and yes, and no,
it depends what you think.
What you think?
That reaction where you think
reaction time is coming from.
So if you think reaction times have
something very directly to do with
something like computation time,
then then yes,
these are scripts will spit out.
Kind of a predicted reaction time
that basically just corresponds to.
So here like I'll just show you
it it the reaction time basically
just correspond that that are
predicted in these models.
More or less just corresponds
to convergence time, right?
So for instance,
like this converges really quickly and
so the action selection would be fast,
and so you expect you predicted
faster reaction time.
Whereas if I am, you know,
if I were to go down here
for example and make.
You know, if I make,
let's say.
Yeah, I wanted to walk
through this more slowly,
but if I make the transition precision,
you know instead of making
it identity matrix.
If I make it like you know,
like I did in the example in the paper .3.
.7.
.3 you know if I just do something like that
and then I run it then it's harder, right?
'cause the the thing doesn't
have nearly as precise of priors.
And So what you get is.
And this is just like what I showed.
There's a much slower convergence, right?
So you should expect longer reaction times.
So. So if that's what you think is
generating reaction times, then yes,
RRR scripts do spit that out,
and you could fit.
If you wanted to come,
I've never really tried to do that,
and I'm not amazingly confident that
those particular reaction times would.
Would you know if that's all?
I'm not sure whether that's all there
is to what generates reaction times,
but like I said,
it's not something that,
as I know that I know of anyone really
like tried to empirically test, you know?
So there's a couple ways you can go about it,
right?
I mean, one thing is you could just fit the
behavior and then once you fit the behavior,
then you could take them.
Predicted reaction times and
just see how well they correlate.
For example,
with the actual subject reaction times,
I mean that would be fitting.
That would just be kind of a a
test of whether computation time
is a good proxy for reaction time,
but I'd probably want to try that first,
but but you. But you could.
Technically we have the predicted
reaction time,
so you could you could definitely
write something up to to fit it to
reaction times instead of behavior or.
Addition to behavior,
it's a yeah you'd have to tweak
the scripts of Fair amount,
but you could certainly do it.
Thanks.
Uh, OK, and that's that's it.
That's it, OK? Alright, well let me
just sort of walk through this thing.
It's it's clear it's clear to
people how to build models.
That's so, so like like I said,
with that free time.
Every time points right it's the
participant starts. Participant,
chooses to take the hint or or two.
So three time points into actions, right?
So it starts the agent has to
take the hint or make a choice,
and then the agent could
take the make a choice.
So again, so that's why we say t =
3 and and then in this case the.
So if you remember I mentioned that
when you're we have both the generative
process and the generative model,
and so we use capital letters.
You know, like big,
a little A and also big D little
D for initial state priors we use
the big letters to correspond
to the generative process.
And if we want to make a separate
generative model then we use the
little like little D in little later.
Correspond to the generative model,
so this thing is big.
D is just saying that the the
true state essentially is that
100% probability is that the
left machine is better, right?
So that's the true generative process?
Or just for convenience,
we're just specifying here.
The left one is the correct one,
is always the left one.
But I could easily change this to make
this point 5.5 and then when I run
these simulations it would just kind of,
you know, switch back and forth randomly.
And then the second state factor is
just the one for choice, you know.
So we just say D.
You know brackets 2 for
second hidden state factor.
And this is just, you know.
Start take the hint,
choose left shoes right.
And so we just put a one here because the
agent always starts in the start state.
So there's zero probability it starts
in any of the other states in the task.
And then here, like I said,
we have little D, which is the
actual beliefs and generative model.
And so we specify the concentration
parameters here.
And so we just give it very
low concentration parameters.
Fat flat belief.
So just point 25.25 that left
or right is better so you know
when this gets softmax.
It's just point 5.5,
but the .25 to mean that it's very
unconfident in that distribution.
Whereas it it fully believes it's going
to start in the start state as well.
'cause that's just the task.
So the next thing that we want to do is
we want to specify the actual a matrix,
so the likelihood mapping
between States and outcomes,
so probably about constant States
and the A matrix and and this is
where things sometimes can get a
little bit confusing for people,
because when you have two state
factors then each combination of state
factor values generates a pattern
of observations in the likelihood.
So what we're saying here is we
just start out saying for all
states for all the states in
in the state, factor 1.
And say factor two,
we want to assign probability
one that there will be no hit.
So the column here is left and right
state or left better state and write
better states were just saying for
all of them they generate no hint.
To start with.
And because 88 brackets 1 here corresponds to
the outcome modality associated with a hint,
so the first a corresponds
to the hint outcome modality.
So it's gonna be 1A for each outcome,
modality or observations.
That possible answer patients.
But then what we do is, we say,
just for the A1 where the third
dimension is 2 and this is what
corresponds to moving to the hit state.
So so this is essentially the left
and the right left and the right.
Left when left better.
Write better states when the agent is
also in the hint state 'cause that's two,
right?
So that's the same as as that.
So first we just say it's all.
It's all know him,
but then we replaced the second one by
this matrix where we're just saying the
probability that the hint is accurate.
Essentially.
So if pH A is one,
then we're saying it's 100% probable
that if it gets the left machine hint
that that corresponds to the left
machine state being better state.
Uhm, you know.
Whereas if I make this like .7,
that means that the hint.
It you know could still be
wrong sometimes, right?
There's only a probability .7 that
if you get the left machine hit,
they'll be in the left machine
state and then the second a matrix.
So for the second output modality,
that's the wins and losses.
And we do a similar kind of thing.
So for the first two values.
So for I = 1 and two,
which is the start state and the
hint state you always observe and all
right 'cause you only observe a win
or a loss if you've made one of that,
you've made a choice.
Uhm, so then here for three.
For the second state factor,
that's the one that corresponds to
choosing the left machine, right?
So that's that's that guy, right?
So this choose the left machine.
And then and then we're saying,
OK,
these are the probability of winning
if the left machine is better or the
right machine is better if you go to
the choose left state and so probability win.
Here is .8.
That just means,
for instance here if I choose left
and it's the left state right?
Then my chance of winning is .8,
whereas if I choose, right?
And or whether or if I choose left,
but I'm actually in the right better
state then my chance of winning would
be 1 minus point at which would be .2.
So in other words,
it's way less likely to win if
you choose left and the right
state is the true state.
And then we just say the reverse for four,
which is corresponds to the
state of choosing right?
And then this final one.
Like I said,
it's all this is is it just says
that for each state it's in,
it observes that it's in that state
and all that does is make it so it's
fully confident about what choice stated.
Since there's no uncertainty
about what it chose to do.
And that's just that's just avoid
certain problems with the agent
potentially not being confident
about the choice that made but
inrets realistic 'cause in real
life we observe what we did, right.
That's just something that's needed.
And you know, if you wanted to,
you know we shall.
You could also specify concentration
parameters for a as opposed to just for D.
You know,
like we did and and there's different.
You know,
we show different ways that you
could do that if you wanted to
model learning reward probabilities,
or if you wanted to model learning
like the hint accuracy, right?
There's ways you could do that.
So now the next thing right is is
is the B matrices the transition
beliefs and so here we're just
saying for the first one, right?
So this is the left better
write better state factor.
We're just saying the left better
one always stays the left better one.
Always you know overtime in a trial.
In other words,
it doesn't change right at time one.
It's not going to switch and the
right one is going to be better at
time two or something like that.
So that's just an identity matrix.
And then like I showed in the talk,
you can just specify for B2.
For the second hidden state factor,
just each action right which just
basically says if I choose this action one,
I'm always gonna move.
So this current state is on the
columns and the new state the
future state is on the Rose sister
says no matter what state I'm in,
I'm going to move to the
start state that's actionable.
No matter what state I'm in,
I'm going to move to the hint state.
That's action action two and so forth.
So you just specify those.
And then the next thing to do
is to specify your preferences,
see So what the preference
distribution looks like?
So weighted down here is for the
first observation modality for the
hints and for the observed behavior.
For these we just said the agent
has no preferences, right?
So it's just flat the only one where it has
preferences is for the wins and losses.
So we initially set those to zero
just to set the dimensionality.
But then we replaced that down here and
what we say is is just like what I said.
So before in the talk at time one
the agent has no preferences.
At time two the agent has a
negative preference.
For loss and some preference for winning,
and that's we're just saying this is
the negative preference for losses,
so it's a negative one,
and the and Rs the risk seeking parameter.
The preference for getting the reward
is whatever number I set up at the top.
So if you remember,
I said it as a four,
then that means it's four at time
.2 and it would be 4 / 2 at time
.3 because the the dollar amount
goes down from four to two.
If you take the hint.
So that's just that's just.
You know how you?
How you do that, uhm?
And then the last thing is
that we actually include.
Pretty sure is is just to
specify the policies, right?
So here we just say there's five
possible policies and there's
two different state factors,
and so you're just specifying
the sequence of B matrices.
Essentially, the sequence of state
transitions that correspond to each
policy where time is on the Rose.
So for state factor one where it's just
whether it's left better or write better.
Agent doesn't have any control over that,
there's just one B matrix that's
just an identity matrix, so this is
just all once but first a factor two,
which corresponds to the choice states.
We give it five possible policies.
One is just stay in the start state,
a time .1.
Or you know at both time points one is take
the hint and then choose the left machine.
This ones take the hint and
then choose the right machine.
This ones choose the left machine
immediately and then go back to start.
Let's choose the right machine immediately,
go back to start.
Right, so you can see.
So each policy is just a sequence
of two actions, which is,
you know what we show over here,
and so that's specified in a variable D.
And then you know if you want, you know.
Like I mentioned,
you could also include habits you know,
so there's just it's just a
vector where there's just one
probability value for each policy.
75 here because there's five policies.
And you can also include literally.
Then it will also learn habits overtime,
right?
The dish layer parameters for
concentration parameters for the habits.
And then here's just a bunch of the
parameters that you can select.
So add A is the learning rate,
Omega is the forgetting rate,
beta is the prior on gamma on the
expected free energy precision.
Alpha is the action precision.
It's a couple of these others
that I didn't explain,
but I mean the the scripts
include a lot of parameters,
but this involves simulating
earpiece and this involves.
Uhm? How you?
Yeah, basically basically it's just.
It's just for simulating certain aspects of
physiological responses like a time constant,
but but anyway,
I mean we we explain all of these in detail
II literally like never use some of these.
And we explain some other ones as well.
But then at the end, you know,
once you specify your model,
you just assign each of these
variables into just into a structure,
just call MDP.
I'm including all the parameters.
And then you know we can add labels
and everything to make it look nice,
but at the end of the day we have
a function that you feed it into
that just checks it to make sure
the model structure is consistent,
and then we have this function right here.
The VB underscore,
SP MDPV underscore X tutorial script
which actually runs these simulations
and then we have figures plotting
scripts here that actually just
spit out those plots and so that's.
That's kind of the kind of you know.
Fairly quick run through of you,
know what it takes to build a generative
model for this task so you know.
So for any task model you want to build,
right? You just kind of want
to think through what are what.
The correct generative model structure is.
You know what are the
different beliefs that the.
Agent needs to have to make a decision
and one of the different tasks
simulator the different observations
that it needs to form those beliefs
in order to make those decisions.
And usually for most tasks it's fairly.
It's fairly simple, you know, like this is.
But so, so yeah, you know.
And then after that, like I said,
all all the rest of it is just,
you know, so here, right?
We just, you know,
we tell it to simulate 30 trials in a row.
And then you can just run the
same same VDX script on it and
it will run multiple trials.
You know,
here we set up the reversal learning where
the left state is better for the first,
like 8 trials and then the right
and then the right machine is
better for the rest of the trials.
I think if there's anything
else down here worth showing,
all the rest of this is is for model fitting.
So for model fitting because
we use dynamic causal modeling
scripts for doing the fitting,
we specify them in a TCM structure
is a TCM dot MDP and then you
can just specify the names with
parameters that you want to fit and
then this is where you feed in the
the actions and the observations.
Are you saying that you and then this
is just like the labels that are used
for TCM and then your actions or TCM dot Y?
And then we have this script called
estimate parameters that does the
actual does the actual estimation
for you and I'll just open that to
show you what's in that 'cause.
This is kind of important,
so here what you'll see is you can specify
the prior variance for your parameters.
I wait to set the same one for all
but you could set different prior
variances for each parameter.
You want to estimate 2 and then here we just.
This is just some kind of convenient
code where we say OK for alpha.
We want to set our prior value for alpha
to be 16 and we just we just log it
because if you put it in a log space
then it'll stay positive during estimation.
Same thing here,
we want our prior on beta to be one.
You know we want our.
That's the loss of version one which we
didn't really use for risk seeking right?
We set up prior on that for five.
And again the prior variance
we just said is 1/4,
and that's just that's a pretty
precise prior variance which
just kind of deters overfitting.
But let's say you just set those priors and
and you know if you want to add parameters,
then you just add things you know to these.
This is just what converts them back.
So exponentiate them out of log space
or out of logic space if it's a value
that it has to be between zero and one.
And then this is probably the
last thing worth showing up.
Is this is the actual log
likelihood function,
so this is where you're feeding in
the probability of the actions that
the agent took under each model,
and you're just adding it for
each for each trial, so like.
White balance after generate
multiple for that.
So if I. If I generate and this is,
this is another thing and we show this.
We we have a table that describes each of
these variables in the in the tutorial.
But if I am, but so if I run some equals 2.
Then what it will spit out after
it goes through the PBX for
simulation is it will set up.
It will it will spit out this
big MDP structure that has a
bunch of different fields in it.
And again like this is just time policies.
A matrix B matrix C matrix
you know D vector etc.
But the outputs over here include,
you know, beliefs about states,
beliefs about observations.
This is the free energy.
This is the expected free energy.
You here is the actual actions
that were taken up, you know,
and so on and so forth.
Some of these are the prediction errors
the you know expected prediction errors,
and here's the simulated
reaction times at the end,
so will spit out each of these and
so basically what the log likelihood
does is it just loops through,
you know,
the action probabilities in here
and then tweak the parameters
until it until it finds the
ones that best match the actual.
Actions that the agent took,
which is in which is in you over here.
Yeah, which is in you.
So this is just saying, for example,
that the agent didn't do anything
for state factor one, right?
It just stayed in the only
action that could take.
Just because there is no
action for state factor one.
But this is saying that I'm
the first time point.
It shows action,
two for the second state factor,
which is taking the hint and
then three and then it shows the
left slot machine for example.
So, but again, all of this,
all the details of how to interpret
all this sister in the tutorial with
the table that explains each of them,
you know.
So if you're really interested in
learning the details and using this,
then you know, really,
you know everything should be in
here to do everything you'd want to
start to finish to start using it to,
you know to build your own model
of a of a task and fit it and
use it for further analysis.
So I guess you know I'll just
stop there and people can ask.
You know people can ask questions
and I'm happy to go through,
you know more of the stuff
in the code in more detail.
If people you know people aren't clear about,
you know how you might use it.
And and like I said,
I I sent the code,
the script or the the link to
the Python version.
We don't have code ourselves as part
of our tutorial for the Python version,
but it ought not to be that hard
to translate if you're fluent
in both Python And MATLAB.
Top note you said is it OK to
post your email address on our
website if people have questions,
contact you directly.
Yeah, no totally so.
So at the at the you know top here I come,
you know so that's my email.
If you have any questions and also you know,
here's my Twitter handle.
So I mean,
you're also welcome to just DM
me if that's easier,
you don't either either way,
but but yeah, I mean so you know
questions anything people want to
know more details about. I know.
Obviously this was like a lot,
so I wouldn't expect everyone
to have been able to.
This is just so impressively comprehensive
and flexible and really powerful.
I'm really excited about diving into myself.
I'm so. Yeah, I appreciate it.
Anybody have any questions
who might be in the audience?
Will be entered come out maybe later
because I I would need some time to.
Yeah, when it's time to digest that stuff.
So alright, well Ryan,
thank you again. Wait a second.
There is a question I think. Oh, OK.
Oh no, that was just thank you.
Just like you.
Oh sorry, it looks like you have a wristband.
Why is there data?
Is there's a toll like 'cause?
There's a likelihood precision,
Oh yeah, so so OK yeah I can I can
show you like how you would do that.
I mean 'cause The thing is this
parameters like that or just things you
would build into the model yourself.
So for example all you might need to do,
like for instance if you want to
include like something like azeda.
You know, say, here, right,
you want so this is this is your
precision for the a matrix for
the hint state factor, right?
So I mean literally in a sense when
it's just this little two by two,
like this PHA is essentially as a right.
It's just controls the precision of
this likelihood mapping in a if you if
if you have a a bigger a matrix and you
want to control the precision of it,
because remember each each
column in the A matrix.
Hosta sum to one so each column
is a probability distribution.
So if you if you have a big array matrix
then you can't just do this kind of little.
You know value 1 minus value thing.
Then you can just pass the A matrix
through a through a softmax function
with a temperature parameter and
the and later then just becomes a
temperature parameter and I can.
I'm happy to send code for that if you
if you want it. I mean we have it in.
We have that kind of thing in in
the models that we've included.
Supplementary material and several
of our previous patient.
That's how I did it when I did it for
my previous work, so that that's fine.
I just wanted to see if there
was any different now.
Very cool OK good.
Yes, I think that's that's alright.
Thank you so much and really cool to
see this being presented so excessively.
Great stuff.
Thank you.
Thank you Ryan.
So just want to say that I will post some
this editional information on your website.
This extra papers and Python code
as well as your Twitter contact.
Yes, I don't know. I don't you think so?
Yeah my man, any anything you want?
You know, the more, the more the better.
I mean you know the whole the
whole point of this whole.
You know both program is just to make this
stuff accessible to a broader audience.
I mean, just,
you know the active inference letter
is just kind of like notoriously opaque
and difficult to break into, you know.
So the whole goal was, yeah,
it's just to make it so a broader.
It's available to a broader audience.
It's great.
Alright well thanks Ryan and
thank you to everyone for coming.