Subscribe
Share
Share
Embed
Recommender Systems are the most challenging, powerful and ubiquitous area of machine learning and artificial intelligence. This podcast hosts the experts in recommender systems research and application. From understanding what users really want to driving large-scale content discovery - from delivering personalized online experiences to catering to multi-stakeholder goals. Guests from industry and academia share how they tackle these and many more challenges. With Recsperts coming from universities all around the globe or from various industries like streaming, ecommerce, news, or social media, this podcast provides depth and insights. We go far beyond your 101 on RecSys and the shallowness of another matrix factorization based rating prediction blogpost! The motto is: be relevant or become irrelevant!
Expect a brand-new interview each month and follow Recsperts on your favorite podcast player.
Note: This transcript has been generated automatically using OpenAI's whisper and may contain inaccuracies or errors. We recommend listening to the audio for a better understanding of the content. Please feel free to reach out if you spot any corrections that need to be made. Thank you for your understanding.
I continue to get attracted by use cases that have more of a social role, of a long-term impact.
You don't have the dataset, you don't have 100 other researchers working on this.
They can give you their point of view. When you don't have ground truth, when you cannot run this, I don't have crowdsourcing platform for children, like I said, let's see, I'm going to recruit them.
So when all of that goes away, and some of the metrics you can use, and some of the algorithms you can use, when most of that goes away, how do you start building again? And I think that's what keeps me very curious.
It's also very interesting to see how children react to the recommender.
We did a tiny user study in the classroom, and we gave recommendations in general for materials they were looking for solving a homework.
And then another one we recommended, and we explicitly put a dataset by the teacher.
And the one that it was just a recommendation, they ignored it, and they went and searched, giving them a source versus not making a difference whether they would use the resource or not.
And I did not see that coming.
Sometimes people don't make the connection, they're like, okay, another type of user, and then you mention a little bit of like, but the evaluation, but then, and then you get that moment, and then people have opinions.
There's no one size fits all, that one is kind of my favorite.
And so then, even if we have enough of a driving patterns, who are we not representing there?
And that's the challenge, because what you expose or not expose to as a child can impact how you develop later on.
We also don't want to be a sensor machine, because that's also not the point.
Trying to say, no, nothing is for children.
So at the end of the day, the recommender is not the parent, that's not the role of the recommender.
We can hide under the sun and say, they won't use it, and they're using them anyway.
Hello and welcome to this new episode of RECSPERTS, recommender systems experts.
For today's episode, I invited Maria Soledad Pera to the show.
She is associate professor for the web information systems group of the faculty of electrical engineering, mathematics and computer science at Delft University of Technology.
Previously, she has obtained her PhD in computer science at Brickham Young-Un University, and she was also previously an assistant professor at Boise State University, which she left later becoming associate professor.
And she is also the co-director of the people and information research team alongside Professor Michael Eckstrand, who, as you might know, has been my guest in episode number 16.
Soledad Perat is actually focusing on information retrieval with an emphasis on non-traditional populations.
And this will be also the main topic for today's episode, where we are talking a lot about children and recommenders and how recommenders can support children in their decision making process.
And what are the distinctions, what are the complexities that go along with that process, and what we need to do different or what developers need to do differently.
So Maria has actually published a lot of papers at conferences like UMAP, of course, also at RecSys, ECIR, CHI and other conferences.
And she was the general chair for the recommender systems conference in 2018 and program chair for the UMAP conference as of this year.
So I'm very happy to have you on the show.
Welcome, Maria.
Hello, and thank you for having me.
This is going to be such a treat for me.
Thanks for joining.
So just for the first moment, what is actually your main name that I can call you with?
Is it Maria or is it Soledad?
Please call me Soledad.
Yes, I'm Maria Soledad Pera.
That's what you see on all the publications and stuff.
But in Argentina, every girl is called Maria something.
And so very quickly it turned into Soledad.
And once I moved to the US, it became Soledad.
That was the shortcut.
And that's what everybody calls me.
So please call me Soledad.
Great.
Perfect.
So welcome to the show, Soledad.
Thank you.
Yeah, I've mentioned quite a couple of things already, but you're the best person to know yourself.
So can you introduce yourself to our listeners?
I'll try to do my best.
Let's see what is the highlights when I share.
I think I will start by you know that I'm originally from Argentina and that's what I did all my undergrads.
And so my computer science background, my bachelor's was in information systems actually.
And that came from from Argentina.
And after that I moved to the US to do my master's.
My master's was in information retrieval with an emphasis on search.
That was kind of the focus.
And actually search for libraries and the use of tax.
And that was the connection to recommender systems, which was the topic of my PhD.
And it was focused on content based recommender systems started with all these tax and false sonomies.
And so in the moment when everybody or most of the other recommender people were working on color at the filtering, some matrix characterization and the like, I was slowly building with some other groups.
Focus more on content based recommender.
So I, the general focus of my master's was on content based recommender of reading materials.
And the last bit of my PhD, one of the user groups of focus was children.
And that was kind of the beginning of all my, I guess, academic trajectory moving forward.
I continue to work on both the search and recommendation side and particularly on children.
But I make the focus of my work moving forward after my PhD on non-traditional populations.
And so while I sprinkle on other user groups, the more heavy focus has been ever since on children.
Okay, I see. So yeah, this was also something that I was coming across when I was reading your bio.
There's a lot of publications that are centering around children or around the work that you published along with your co-researchers.
However, on the one hand side, it says non-traditional populations.
And on the other side, there is a focus on children.
So just maybe to lay out the landscape of non-traditional populations, who can we think of when reading non-traditional populations?
So I think for me and all the group of students and also other colleagues that we have been working on this for a while, the idea of non-traditional users, we think of what you wouldn't expect the bulk of the research to be targeting.
And if you go to the RECC proceedings or now SIGIR also has a lot of recommendation system research, you see that most of the bulk of the work and also the bulk of the data set and the algorithms, the recommended algorithms this time, are kind of driven by this grown-up population, like 18 onwards.
English speaking is very heavily the target.
And so when you start thinking, OK, what happens if you remove that?
If English is not your first language, if you don't have that level of traffic, if you might not react to the content suggested or presented by these systems in the same way, who could be these users?
And so as long as you start finding children is the default, at least it was for me, but you start thinking people affected by mental health issues, people that have low levels of literacy and therefore might need more from the recommenders to be aware of that would serve them, people that have cognitive disabilities, different use cases.
So, for example, one of the use cases that it for me is also interesting is like teachers that they are receiving recommendations, but they're not necessarily for them, but then to be using the classroom.
And so this third party thing is not what we what literature considers the traditional user, which is the one user or at best the group of users, but they have the same goal.
We went through many names.
It was like niche users, but I from early on, I didn't really like it because I think it's just their users.
That's just not the expected. So some people like to call them then their serve users.
I think they're served. They're just not in masses, sir.
So I would like to that attention. So for me, they're non-traditional.
Yeah. OK. OK, I see. When first approaching this topic of non-traditional populations and reading a bit through the material that you published, I was coming across from time to time that it says there is the assumption that systems that we design or that are, let's say, dominant in the RecSys landscape out there are mostly tailored to adults and therefore not that much suited for children.
And therefore we need to make sure that they also cater to children's needs, which are different and so on.
My question is, so if we model users in our recommender systems, wouldn't we just model them in a way that they also cater the, let's say, characteristics of, let's say, adolescent users and not only of grown up users?
Or why is it that we need to build separate, maybe separate is the wrong word, or special systems for these users?
That is a great kind of way to fix it because for me, I think that there are two sides.
I think it depends on the use case and depending on the type of material that you're going to recommend.
It might be that you necessarily need a completely kind of different way of approaching the recommender process.
Sometimes I'm thinking of the reading.
If you are recommending materials that is kind of about reading, it could be whether for leisure or to support kind of their learning, then there's a lot of more aspects that go into it that need to be taken into account.
Like, for example, the readability.
The content might be the right content and it might be the suitable content and there's nothing wrong with the traditional recommender.
But then if the person on the other side, and it could be a child, but it could also be someone with low literacy, cannot read it.
And so then that's a dynamic that we cannot just pick up from patterns.
This like, what is your reading level?
It's not something that can like suit it from the parents.
It's something that needs to be very specifically modeled.
Now there are other cases like it's a movie.
I think it can be picked up from patterns.
And so you could use the traditional recommender, I guess, algorithms or the traditional recommender framing to adapt to different type of users.
Totally in agreement.
The issue with that, though, if we go back and say, do we have enough information to get to know these user groups?
Most often than not, the answer would be no.
If you look at the type of data sets that are available for research from the big movie lens to all the songs ones or the good reads, primarily they're driven by interactions by grownups.
Maybe some late teens.
If we're very, very lucky, we have been lucky sometimes.
And if we're very lucky, we have the demographic information.
And so, okay, if you have that, that can drive a little bit more of an analysis.
I said, okay, we use whenever possible these big algorithms that have already been proven and really researched to know to work.
Because it's not that I don't want to.
And when I say I, it's like all of us that are being working on this.
It's not just me.
It's just it's less feasible, but it's not just me.
It's not that we don't want to use it.
I think whenever possible, you want to leverage all that knowledge.
And that's why we would like to this topic to be more broad, because then you could leverage more of what the as a community, we know about this.
The issue is that it's not really available.
And so sometimes we just cannot for really young chingler.
We're never going to have the ratings or the traffic unless you are a company.
And even then it's a very fine line.
And I'm not inside of the company, so I wouldn't know.
But it's also very fine line.
How much you can use that data or how much you should use that data.
Yeah.
And how much that data is really a user versus the parents that put certain constraints or the care given.
In the case of other users.
And so it becomes this.
Is it really you when I try to explain this to my friends, what I do, people ask like, OK, but on Netflix, you have the children.
Yes.
But then is it only the children that are picking the thing or is the pattern that is putting the children.
So it's always that dynamic of it's really the user, the individual making all the choices or that a lot of constraints that because it's the different user group need to be taken into account.
If the parents have certain boundaries, whether the child likes it or not, maybe it's not suited.
And so then it's the what they want versus what they need versus what they should have, which when you're a grown up, you make decisions.
You don't like the recommendation.
You move on.
It's not really harming anything.
So I think that's the case.
And so whenever possible, sure.
A lot of the times we try to start with something popular or with the traditional more recommendation strategies and then filter out.
That's usually a better way to leverage everything we know.
And then we say, well, once we get the top 20, top 50, can we start cleaning up and then reprioritize what is more fitting to the target audience?
OK.
That's kind of the bulk of the work that not just mine, but the working with this non-trational users that it's out there.
I see two very interesting aspects in what you mentioned so far.
So one of them is actually that we need to specifically model those user groups.
And the other thing is actually that there is a lack of, I would say, transactional data on these specific user groups.
So data that we could learn from to also understand better what are the differences and in what manner we need to develop different systems suited for those needs.
And you have also already touched on that harms that might go along with the application of recommender systems.
Let's definitely come to that later.
Also the multi-stakeholder aspects, because it's not only about the kids, but it's also about the people that are responsible for them, mainly the parents, but also the teachers or other people, if you think about an educational context.
Let me start with a follow up question regarding modeling, because this is still something that I'm not fully getting.
So let's take the example of an educational domain.
So something, let's say, where you are going to recommend books.
I guess there was that paper from the recent U-map that you have also been an author of, which was covering covers, characterization of visual elements regarding sleeves.
And in that paper, you were basically researching on content representation for book covers and in what sense these book covers differ in certain characteristics across different age groups.
And there were these five age groups that you used that were according to the reading development stages.
And this was actually where I was kind of getting very much interested, but also comparing to the adults, because when thinking about an educational context, of course, children are generally different from adults, because adults have gone mostly through scholarly education and learned how to read, how to comprehend information, how to distill what they need from it.
However, there might also be other domains where even adults are still learning or progressing.
And there I'm still thinking about, can't we model adults and children in the same way create models or representations that are complex enough to capture the skill or the level of comprehension that people, regardless of being young or older, bring to the table to comprehend content or to acquire new skills?
So in which sense do these models for users need to be different or don't they need to be different at all?
Too many thoughts.
I think the short answer is yes, if we can really think about user, if we can really put the user at the center, and this is not one side, like kind of one blanket statement for all the domains.
There are domains where you still put the user at the center, but you can learn a lot from the interactions.
But in domains like, for example, education, I do think that there are some factors that can and should go into this design of this user model, and then it shouldn't matter if it's a young child or it's understanding certain constraints that can and should impact the final outcome of the recommendation.
And so if we really can model and understand that well in the context of education, then I think it would apply to a child, it would apply to a teenager.
Now, the restrictions would be different because the things that we say yes or no or the requirements for the child might not be the same or not by nobody, they won't be the same for the young child, whether it's or the teenager or an older adult.
But it could also be for a more grown up that has different literacy levels or different skills, or that are learning something new and therefore don't have the background, just like children are picking up background about life and the environments surrounding them.
And so I do think that at least with my line of research and the line of research of my colleagues that have looked at the concentration population, they all have always been to try to model users.
And we always kind of argue for this idea if we get to know these users that are unmolded by the environment, then we can learn a lot and then takeaways could really impact the recommendation in general for other type of users, not just children.
And so we have been trying to put the user at the center from the get go.
It's just that now it became very fashionable to do that, to really think of the user first, rather than the interactions. Then yes, you take advantage of the interaction.
So I do think, particularly in the education domain, that being able to understand the different perspectives that made up this user and the different needs that made up this user, then as long as it's adapted to the different type of user and dynamic, and that's another thing.
The requirements change, I think, much more faster than they would like. Your taste in movies, yes, they change. And so we know that there is already work on how dynamically change your taste, sure.
But your level of how you read or how do you acquire knowledge on a topic, that's kind of much faster and you move on kind of quicker there.
And so as long as we can model the dynamics of that and really understand that you might learn more about the topic, but maybe not so much on the readability or vice versa.
If we can model all of that, then I think it wouldn't matter if you was a child or a grown up or different perspectives.
But I do think that that makes it for a very interesting user model that is very human driven, that it requires understanding human factors beyond interactions that can be picked up just by us computer science researchers.
And I think that that's where it gets very interesting and it requires a multidisciplinary perspective.
I have done things when at the very beginning, like I run experiments or with my students, we run experiments and we do recommendations and then we're like, we don't really get this.
And then you talk to a teacher and it was like, yes, it's because of this. And I was like, we're missing that. And so then you understand that it's not just bringing the others to help you understand the outcomes, but bringing them from the very beginning.
And at the very beginning I said, what do we need here for this context to be meaningful? And it cannot just be a computer scientist perspective on all of that.
Okay. Okay. I see. Really find that aspect of dynamics really interesting. Since I would also say this is a very general problem that we encounter in recommender systems.
You already said user preferences can dynamically change. I mean, they are also very context sensitive, which is there even more frequent change in the preferences driven by the context in which people find themselves.
However, I would definitely say the dynamics, not only present with regards to the preferences of users, but as you said, also with the development of the users when we think about children, because I mean, they are developing much faster than adults are developing.
So maybe the user preferences say change quite dynamically in both, let's say populations, but the skills and the aptitude to process information that is presented by a recommender system is much faster in adolescence than it is in an adult.
And this is something then you would need a model to kind of model this aptitude of users to then use this as a signal, right?
Totally. And I think that the skill set is what it makes it so interesting, but also what makes it so complex. One of the examples that early on, because when I started for whatever reason, along with, no, for whatever reason, the last paper in my PhD was on book recommenders for children.
And readability came up being this thing that needed to be there. It was non-negotiable. And from then on, even though that's not my area of the one that I do the most research, I continue to try to understand readability more because it goes back to everything content based that I do for all the traditional populations.
But my point with readability was, okay, the first time around, I was like, okay, I model readability. And at least I said, okay, you're the first grader or the fourth grader. We map the readability to that.
And then the more you start chatting with teachers and literacy experts, they were like, well, but on the fifth grade, there are children that read about their level and below their level.
So then you cannot just say the fifth grade. And so then as a starting point, you do because you need to start somewhere.
But then you start saying, okay, how do I try to understand the level without asking because you cannot just ask each other, which level are you because they don't know or they can say, so how do you start picking up those signals based on what they read in the past or whether they struggle or not or how.
So from different perspective. But then the other part is, for example, in the recommendation, if they like, I don't know, a topic, bicycles.
I mean, the Netherlands, the bicycles are lower. Let's say if you like bicycles and you're interested in bicycle, maybe you are reading about the content of bicycle at a very higher level because you care about that topic.
But maybe for everything else, for leisure or for learning, you're not. How do you model the two things? Because it's already challenging to model the one.
And on top of it is the world. By the way, if it happens to be this topic, you're more advanced. We haven't gotten there. Or at least I haven't gotten there yet.
But those are all these. And I think that's why I keep working on this particular user group. And I keep going back because every time I think I advance on something, I feel like I'm making progress in another pop up.
It's like, oh, by the way, now you have readability that is not by grade, by surprise. If they like a topic, they're going to need something different.
And so that's what kind of keep me going because there's something more to model about these users that if we can really understand it, then we can start leveraging all these other recommender algorithms and mix and match what we know from the bigger research and what we know from this one.
But I continue to find it fascinating because of that.
Definitely. You have already touched on a couple of examples, especially from the domain of education and education oriented recommender systems.
Also, we talked about book recommenders or book recommendations.
What would you see as the main prevalent domains in which this research on on recommenders for children plays a role or might have an impact?
So what where do you think this is a very high importance in which domains for for recommenders with maybe different also harms associated with recommendations?
Such a challenging question.
Let's see. I continue to value a lot the need for book recommenders, of course, adapting to times, maybe not maybe for sure.
Children are reading less and less the hard copy of the books and more digital versions.
But at the same time, research also tells how children and I keep saying children, but I mean, like in my definition of children, like from very young to like eighteen, they're reading less and less.
And so this is where the recommend regardless of the device, regardless of the device, they're just the reading for pleasure is not as common anymore.
And so then also a lot of the research on modern on the library side and more on the reading side is this idea of autonomy and finding things that they find interesting, not the reading list at school that says, oh, by the way, here are like 10 things that you.
And so I think this is where recommenders could have a fantastic point where it says, hey, look, these are things that you are going to like.
These are things that match your skills. So you're not going to be off footed by maybe it's too challenging or these are things that map to everything you like and you enjoy.
And and I think that that has like a bigger impact on society that yes, a movie or a toy that you might recommend that you would also be super fun.
But that this idea of finding that matching moment to say, hey, maybe you want to read more.
And we all know that when you read them, that has an impact on on everything else you do in life. Children are not seeing that and they don't want to hear them read because it's going to be good for you in 20 years.
But if you kind of started with getting them engaged. And so I think that a book recommender could be could have a social role.
So I see the tremendous value of that. And again, for children, but also for people that have low literacy, finding the right thing.
And that could also be kind of biased by my own experience. Like I grew up speaking Spanish. And for me, it was a human recommendation or a recommender.
But I wasn't when I started learning, I wasn't reading in English because all the I was a grown up learning English.
But I keep getting all these books that were like the photos, whatever the enchanters like I was a grown up.
And then a family member gave me a simplified English version of Pride and Prejudice.
And I started reading in English because of it. And it cannot change how I engage because that kind of had, yes, the things that I could read the level.
But it was also a topic that as a late teen, I actually care about all the Shane Austin literature.
And so for me, that could be that recommender could have that massive impact and say, hey, let's see what you're at in your skill set.
Let's see what you're adding. And let's give you something that you want. But yes, that's just like my very own biased version.
But then if you think about it now, how children are learning and especially after covid with all these teachers or platform, some kids are they learn better from a video rather than from a book or a lesson like I was never that person or by listening to something.
I'm not into audio books and I know a lot of people that found them like fascinating.
And so I just don't pay attention to the audience.
But like and so being able to say, OK, how about recommender that can identify these materials, these videos that are already out there that are great videos, but they have more of an educational value.
And then someone that is not well suited to learning by sitting and reading a book or going through a lesson, they can learn through a video.
And so a recommender could be that bridge to resources, but tuning to this is the level that you would understand.
And this has the right background and it will not be something random that it has the title for it.
But then it will be something completely off that you don't want to because that's the other thing.
As a grown up, I get a recommendation in YouTube that is not what I want to skip it as a child.
Already that impact. Yeah. Whether it's for learning or not can already has.
I don't want to say harm, but it could have more of a I was reading a paper the other day and they were talking about how some videos could be disturbing.
And I don't know. It's a strong word. And because it's not always terrible, but sometimes it could be that it's like that shocking image.
And so the recommender for those users have that mode of buying and you need to have that extra care, that extra layer that for the grown up.
For a grown up or for the traditional grown up, let's say you can escape their grown ups that are affected by different.
Like I mentioned before, cognitive disabilities.
I mean, that held that something flashy or particular can be very triggering.
And in those moments, the recommender has that social role to be that layer.
And how do we make sure that that happens? It kind of gets complex very quickly.
And so that's why I think it's different. And that's why I think it's important to continue to dig deeper.
But then, yes, the recommender, it's all over. But I'm just I don't know.
I continue to get attracted by use cases that have more of a social role of a long term impact.
So basically to kind of perceive the recommender system as a tool to spark joy, but to also, let's say, prevent harm to stay with that term.
And therefore consider it as a tool that carries a lot of social responsibility in that sense, especially if we think about the educational domain.
Exactly. And so that's kind of my take and what I continue to come back to.
But at the same time, it could also just be the fun thing.
But the fun thing with the extra levels that you need to keep an eye on, depending who's the user group.
It's just the thing.
Okay. Okay. We talked about these media based recommenders, whether it might recommend audio books or books to buy or to read, whether it might be on a tablet or as a hard copy or something like that.
If we go a bit more to other domains, which complexities would you see there?
So, for example, nowadays, also children from a certain age onwards are exposed to social media, maybe Instagram or TikTok, the role of social media and recommenders and social media for children.
But also what about e-commerce?
So, of course, to buy stuff on Amazon, for example, you need to have an account.
You need to be somehow eligible, which somehow correlates with having a certain age.
But also children, regardless of the aspect, whether they can create an account or not, they can use the one of their parents or something like that.
So if we think about social media or if you think about e-commerce, which challenges do you see there and what is important there to cater these systems also for children?
That's another, you give me all these like softball questions.
And I think like sometimes, and I'm going to apologize in advance, but sometimes it's kind of hard for me to detach the, I can't give you the core researcher, like the core researcher or the more detached kind of answer to all of these.
But at the same time, these are issues that I continue to read a lot about because they continue to inform what we, our group, we end up doing for research.
And so I kind of also have that human take or the personal take on it.
And it's hard to kind of decouple, but with TikTok, for example, or starting easier, let's say, with Amazon, one of the issues is, yes, children are not going to have, particularly the younger ones, they're not going to have their own account.
And so then if you have these purchases of these different patterns of it's the parent buying for the child or it's the child that they want, or it's a grown up buying for a present, I still get like every time I buy some of my friends, like for their kids, something, then I get all these recommendations, like it was a one.
And so the first thing is trying to really decouple or try to understand who are the users that might be generating all this traffic and, and there are cases with the purchasing pattern that particularly again on children, it sometimes is not enough because you bought this, buy this.
Because this, buy this might be something completely inappropriate or completely unsuitable.
And so the digging deeper in the e-commerce that just because someone bought something, the other person, when they're young, may not come from the different background, may not come from the different, you know, they may not have the same religious or cultural background.
And so then this, buy this, this other thing, the other thing might not be the fitting.
And so that for me is one of the key challenges.
The other way around it, it's also true is the continuous representation of stereotypes.
And so then we always go back to the maturity, even if something is catered into children, it's to which child is catered in because there's not a blanket statement.
There's no one size fits all.
That one is kind of my favorite.
And so then even if we have enough of a driving patterns, who are we not representing there?
And that's the challenge because what you expose or not exposed to as a child can impact how you develop later on.
And so for me in that because of this, that it becomes more challenging for this user group.
And so that's something that we continue to ponder on a lot from the social media.
I think, yes, that's what a commander could help.
And that one, I think it's way trickier.
I was reading the other day a paper and I'm terrible at remembering names of things.
We will find it out.
With a couple of keywords, we can find them about how children rely on TikTok particularly to get their news or to get what's kind of happening in their world.
I haven't played with the recommender enough to get a feeling yet.
But how you can get a lot of good information, but also how you can go into that rabbit hole of maybe that's not what you want it.
But then you're kind of hook into it.
The same with Instagram.
And so how are we looking at that when the user is a different type of user?
There are a couple of papers lately on people on individuals affected by mental health by the university in Minnesota, a TV chancellor and one of my former students, Ashley Milton.
The studies that what they're seeing about mental health and social media and particularly this recommender that keep perpetuating some of the stereotypes or some of the things that are going to the wrong side.
It's very challenging for the non-traditional user groups.
The stereotype is something that I'm playing now with one of my students on what are the stereotypes that this recommender are bringing up to children.
And I think that on a book or a song, maybe it's like it doesn't have as much of an impact.
Maybe I don't know. But on social media, that is so immediate and so fast.
And it's one thing and another and another.
And then once you went into that loop, it's all the same.
I think we need to be looking a little bit deeper because, yes, children and particularly young words are not the target audience for that.
And they shouldn't be exposed.
But that's the reality.
They are.
And so that's kind of the thing.
They shouldn't.
And I think that they're all the safeguards in place so that they wouldn't be exposed to something that they don't know how to handle.
But at the same time, they're going to be exposed anyway.
Yeah.
For me, I always made that correlation with search engines.
We say, oh, search engines are not for children or are not geared for children.
And so then we cannot expect that the articles or everything is going to be well, but they're using them anyway.
So then we cannot need to pay attention because they're using them anyway.
And so I think that this is going to open in this digital ecosystem and this new era, the TikToks, the integrams and everything else that keeps moving so fast, how children engage with material there and the value of recommenders to kind of steer them, at least in something that is within what they can grasp and design a magnetization would be very important because we can hide under the sun and say they won't use it.
They're using them anyway.
That's, I would say, a naive claim and maybe also used as an excuse to not look deeper into how to cater their systems also to children, I would say.
And so you kind of have to.
And this is not to blame, like, the companies or anything like that.
And even the researchers, there's that much that you can know or get access to.
But the idea is that at least we should be having these conversations a lot.
One of the reasons I think you mentioned it or not, I don't remember if I mentioned it before, like the Keep Rock workshop that we at the beginning, we tried to to host at RecSys is try to like build community and try to understand with all researchers and keep this going because it's for the discussion.
Maybe we cannot solve all the problems.
This is too big.
But keeping an eye on it and continue to keep it in the surface and understand more and more what we can or cannot do or what we should do because we recommend that it doesn't need to be a sensor as it's not everything.
It's not a question of like the safe search that it needs to be a gate and say, nope, but understanding what are the leaways that you can have and how can you help make sure that within the leaways or the different perspectives, you're still protecting this dedicated population.
Whatever that dedicated population is, you don't want to just shut down everything because that's also not real life.
Yeah.
So that's kind of not the point.
Yeah.
But at least control it.
I guess you're already building quite a good bridge to data sets and also potential harms or potential downsides of recommenders because you basically said it's not bad intention by, let's say, companies that don't take it.
Companies that don't tailor those or their systems for children or make it possible to distinguish properly.
So for me, it's two sided because on the one hand side, there's a lack of data on these non-traditional populations.
But on the other hand, we basically need data to distinguish and to learn.
However, once we arrive at the point of collecting data and collecting data about or on younger populations, then we are entering, I would say, a very critical area because they might not be able to grasp the full extent of sharing data.
And so, sometimes, you have those interesting downsides that we are researching and working on.
So how can we solve that problem of not having enough data about children, but on the other hand side to make systems appropriate?
I know it's like the chicken and the egg problem.
And I'm with you with the collecting data.
I think that it's a challenge because it's not possible to be really representative of all the skills that even within one classroom.
So whatever little studies we have tried to do.
And if you look at not just on recommender, but on search, should like expand it a little bit more to this idea of how we can do that.
Should like expand it a little bit more to this idea if you try to do rankings and stuff.
Well, you can get a school, let's say a classroom.
Even within the classroom, you're going to get all these different.
And so you want to get be representative, but also how can you make sure?
And so I think that before we start or we even claim that we need to collect all this data for children, which that would not be my first step.
I think we have an option to kind of take a backwards approach and try to model the human.
What do we know about the human?
What are the human factors that we know we can model?
And so then we can model the different perspectives.
And at least if you model that, then we can test it and testing with doing user studies to test with children and see what they that one.
It's easier because it could be control.
It could be in partnership with classrooms or with literacy labs or so.
Then you have that measure of control.
But on the output, after you have an initial sanity check that you're not going to be exposing them to anything.
And so I'm more of the less model the human.
And I think if you look at the different recommenders that are out there for non-traditional populations, whether they are companies, well, again, I don't know the insights of the company or whatever I know is here.
But if you look at let's stick to the research, they're trying and they have been trying for a long time to model the human.
Now, when we hear the eta of AI, it's like suddenly we discovered that that was a human all along.
But if we go back in the research, the past 15 years or 10 years, the human has been there all along.
And so if we really try to understand that human and model the needs and the perspectives and the vision, then that's what we can try to use to get to know more and serve this human better.
But when we do that, then how do we evaluate and how do we assess that also changes?
Because now we don't have these humongous data sets to say, we're going to do offline evaluation and see what you're at.
With Michael and the pirates, we did a couple of years ago now these cool kids paper.
And we have the movie lens data set, a version of the movie lens data set.
Now I mix the movie lens or Lassafem data set.
Now I'm confusing that one with a baby shark.
It's like that's when you like this.
But it was one of those, some very few of the song kind of data sets that had demographics.
And they didn't have children, but at least they have 16 onwards.
And even when you do the 16, 18 and 18, 20, and then you do the bulk and then the higher, you can already see that if you test the recommenders, as in overall, they were working great.
And when you start looking at the chunks, they were not working that well.
And so then if we don't have already the bulk of the data to be able to even pinpoint and discover that, then we need to come up with different ways to evaluate.
And that is also an open problem.
And that's what you bring outsiders in and say, okay, maybe a proxy, maybe before we expose a child, we bring a teacher, we bring a parent, we bring an expert in, I don't know, the music domain, but a proxy that could say, this might work not so much because you cannot just do an initial study when you are uncertain and say, oh, child or old person affected with mental health, let's see if this doesn't trigger you.
It's a lot more than, oh, I don't like the movie.
I'll skip it or say, no, I rate it low.
Yeah.
And so I think it's the thinking about that and how it kind of changes what we know about our starting points.
We already know that evaluating the NDCG, if you're doing top end, the RMSE, we know that they're not perfect because they're not looking at already all the perspectives, even in the best of scenarios.
They give us a blanket number, but they don't look at biases.
They don't look at fairness.
They don't look at the entirety of the population.
But at least they give us a place to start.
Yeah.
What about when we don't have that?
Yeah.
And so that's what, again, it's what makes it interesting, but also so complex about working on non-trivial user groups and working when you don't have data to at least get a sanity check on that feeling of like what's going on.
And I think that that's what going back to the model in the user, that's what we started doing the covering covers paper that you mentioned or the baby shark paper that we did a couple of years ago.
Trying to understand, okay, if I don't have the driver of the child, but I know material that has been curated for them, what is it that this material have in common?
Can we try to figure it out?
And then use that as a signal to re-rank or revisit the recommendation of books, of songs.
I was the one with the song.
That one is funny.
You see little kids singing these very inappropriate songs.
They don't know what they mean.
But they're like catchy tones and they're thinking, well, do you want the recommender to really give the catchy but completely inappropriate song?
Maybe not.
Yeah.
And maybe we can start looking into things like that.
Just because they're catchy and they have this at-beat tone, great.
It matches the pattern.
But is it suitable?
Maybe not.
And it's funny for Ask Rona, see them sing and like, but I would like the recommender to be more mindful of that.
Yeah.
Yeah.
To be mindful of that and to model the user, I would say is one piece, but the other or the one that is associated with this is to really put that into practice considering a real system or the data that comes into a real system, the tracking data, the self-reported data.
I mean, when you say we need to model the user, we are representing users in a system, let's say, as distributions across genres to show us somehow what are their preferred genres.
Also to have some estimate about what books have they interacted with and what are the associated age recommendations of these books to somehow assume what might be the age of a user if they don't report this about themselves.
So even then we need to somehow record transactional data to, let's say, support our assumption of what the user might look like.
And on the other hand side, there is that self-reported data, for example, collected as part of some onboarding process where adults along with their children as the final recipients of recommendations are telling the system what are topics that you love.
What are topics that you like to collect initial texts that fuel a profile or also, hey, what is your birth year to infer what your age is to then see, okay, now we see that you are seven years old.
So we now do know better that recommending to you books that are suited for people from 10 to 12, for example, are not the right ones.
So how do we mitigate these problems?
Because, I mean, of course, you can create a user model, but you still need data to fuel that specific model for each and every user.
100%. I think it's you kind of start taking whatever breadcrumbs you have, I guess that would be the best way that I can say it.
I think sometimes you will have it will be depending on the recommender and the recommender environment.
Sometimes you will have at least a little bit of the onboarding so that at least will put you a little bit closer to what could be a good starting point.
There was a couple of years ago, a long time ago, size like Biblio Nation.
What they do is within you get an account.
And so it's a very protected account.
And the teacher is often the human in the loop in the system.
But children can record what they are reading.
And so parents on education can see what kids are reading.
And it also records like a digital reading log.
And so then you know which books and how much.
And so from there, over time, you can infer more about the specific of each kid and say, okay, these type of books or at this level of reading.
But at least even then you need to have a place to start.
And so this idea of when we try to address the call start problem for in general, there's always a new user from with whom you know nothing about.
And then you say, okay, let's see if they give us at least their age.
We try to say, let's see what other people in the demographic if they give us like nothing at all, at least we go with the popular.
And so we could do the same for this user group, but tailored to the user group.
So rather than say, oh, a movie, if we know that you are the child, rather than give you the popular movies, let's try to tailor it down to at least your target audience.
And then over time, these systems, whether they are like the research systems that we try to learn from or like the recommender algorithms that we try to like learn from.
But also the commercial ones can try to tailor that initial very genetic, but at least genetic and tailored to the user, non personalized by generally, I mean, like non personalized, but at least more tailored to the target audience.
Then over time, you can start to personalize more and more.
But looking at different type of perspectives like the readability, because the genre, the taste, I think that we already have a lot of recommended algorithms that could leverage that.
So we take advantage. But the readability, the suitability part, that needs to be a side thing that needs to be taken into account.
And so but at least it would be a way to have that starting point. That's what we were trying to say with the covering covers or the baby shark.
If we at least know your age, is it going to be perfect? No, because there is not such a thing of like everyone's age.
But at least it's going to put us closer that we just give you the popular books for like all children.
If we already know you're 12 or 13, well, given you like the children, popular book, it's a very big thing.
If we can put it closer, you put us in a better place to start. And so that's kind of the sinking process.
Finding what is that starting point that is even if it's non personalized, or if it's minimal.
And then if we get interactions over time, as a researcher, I'm not going to get those interactions.
I wish I could create the movie lens, the platform for children.
That will be a nightmare of epic proportion, ethical wise, all the checkboxes.
But yes, that will give me a lot of like, richer knowledge about these interactions.
It's also very interesting to see how children react to the recommender.
We did a tiny user study in the classroom, and we gave recommendations in general for like materials they were looking for solving a homework. And then another one we say recommended, and we explicit put it back as said by the teacher.
And the one that it was just a recommendation, they ignored it, and they went and searched.
Rather than using the recommended item, because they didn't know where it was coming from.
So trust and explanation plays an important role.
But I didn't see that coming. And the one that says this is recommended for the teacher, everybody use it.
And very few actually search further because they trusted that source.
Okay. And that one I did not see coming. And it was for the fourth grader type of grades of like 10.
I didn't even imagine the concept of trust being an issue because you imagine this like digital kids that were like digital natives.
And they were like, they were like, the teacher, yes, this one.
It's not that they love to search because they need two types. And yet they went that route.
And so I still like to do this small studies in a control setting within the safe context.
So that then we really get to know these users. And then we try to react to that.
Because now we know that trust and explanations, it's a thing for them that I would have never put two cents on it.
I mean, trust and explanations for recommendations also plays a general role in recommender systems.
I guess there is a lot of research around this.
No, no, 100 percent. 100 percent.
Is it somehow that you have underestimated the impact that explanations have to drive trust and recommendations for these populations?
Yes, that was the thing because I have, yes, particularly on the content base that I continue to kind of like dig deeper into that side.
Explanations and trust are a key factor and particularly trust the explanation, of course.
But the issue of trust and I have worked a little bit on recommendations by the social groups.
And that was for grownups, not for children.
And yet the trust you have on who the recommendation is coming from and knowing like, oh, this friend, it's I don't know, another information retrieval is like me.
So I'm going to trust that recommendation more than if it was coming from someone.
And so, yes, as a concept, it's, of course, 100 percent.
But I completely underestimated, particularly with young children, that they would even bother because and this might be completely biased from my experience with the work on the search side.
This idea of like going always to the first result.
And if the first result have even an inkling of the result of what they needed, they just don't bother with going longer.
And that's just consistent.
And so I was like, OK, if somebody gives them directly the practically the answer rather than go type your query and get.
Yeah. Yeah. Are they going to.
That was my assumption from the search site.
And then it turns out that now that that only was the solution if the new where things were coming from.
And now it's a kind of inspired like we haven't followed up on that yet.
But if they see if they coming from their friends or if he's coming like I don't know if saying it's coming from now, Chachipiti, I don't know.
But if you tell them that, will they trust it the same way?
I think it's super interesting to see if they would or not.
But at the time, giving them a source versus not make the difference where they would use the resource or not.
And I did not see that coming.
Interaction and the interaction between children.
So the recipients or the consumers of recommendations and the system or the recommendations themselves.
Which other aspects have you observed in these interactions when considering that scenario of children recommenders?
So what is interesting or also what is special about how children interact with recommendations from my work?
Very little. But from work from other researchers, I remember a couple of years ago at the kid workshop, we have a couple of papers and consistently they were very visual.
So do you mean the recommendations or the children were very receptive to visual signals?
Very receptive both. Very receptive to recommenders that were very visual in their outcomes that they could interact.
And even when they were making when they were trying to get recommendations with they would say, oh, I like this movie that they could actually see everything rather than less of things like being able to see them.
And so book covers or the covers of movies or even when there was one that I thought it was very interesting to say you could put in front of the recommender environment like an item.
Like I would say, oh, this is my phone and nobody's going to see it now that I think about it.
But I am showing you my phone and you don't care. But like you could put in front of the camera in item and then you would get recommendations like this.
And so and it could have been the color of the shape. And they were very into that.
That in the particularly younger children we're talking about now, these younger children also with the preference elicitation, they were not into typing a lot.
And so if they could click things that they would like that would start is different from, I would say, adults or that was the key difference that I saw from adults.
And so I think that this idea of like, we don't need to say, oh, this one looks like a color that I like and clicking more and more into like the ratings, the thumbs up or down and the review and that gets things going.
And with children, it was more the visual interactions. And that's also something we, or at least early on when we started looking at it for I saw it from other researchers.
For me, it was from the search aspect, which was the same formulating equity, dragging and dropping things rather than typing.
It would be interesting to see how they would engage with recommenders with a vocal assistant or with a conversational kind of back and forth.
I don't know how that would work with children. And I have been particularly interested in that. And I think going back to that, I think that's why the recommenders of TikTok and the social media, why they're so important.
Because I think the key recommenders that children interact more with are those.
They are very visual. You get always displayed videos or photos and so on.
The same with YouTube and things like that. And so I think that those are, it's not that the other ones are not important.
And then, identically, the books and the things are the ones that I feel more attached to.
But I think the ones that have that visual environment that they're immersed with and they're more willing to interact, those are the ones that we should be more mindful of.
I mean, some aspect that drives interaction is actually the suitability of recommendations.
And I'm actually not saying relevance here for a reason, because if we say relevance and we directly think about accuracy again and then we enter that debate around accuracy being the only signal to assess when assessing the suitability of recommendations.
Which by far, and also I think for most of the listeners, is not the only criterion. But of course, it's also not unimportant.
However, when we think about goals and needs that drive the goals of the recipients of recommendations, what do we have to keep in mind for this non-traditional population of children when it comes to the goals that children have?
When being confronted with recommendations. Or of course, I mean, in the end, recommender systems shall support decision making.
So what is the decision making problem that children face that then drives their goal of making a decision, for example?
So this is one thing that is interesting for me and mostly focused on the consumers of information. But we have also said the children as the recipients of the recommendations are not the only stakeholders in that broad set.
There are other stakeholders and you have mentioned their teachers, their parents. Let's tackle both of these aspects.
So what are the stakeholders in the overall setting and what are their diverse goals and how are these goals maybe different from grown up consumers of recommendations?
Another, yes, they're all very deep. They're making me think this morning more so.
I want to start with the different stakeholders because I want to think it's the most challenging and then it becomes a trade off because it's impossible for one algorithm to make everyone happy.
But when you think of children, first you have the children, what they will like. If we go back to the non-traditional relevance, at the end of the day, if they're making a decision, it needs to be something that appeals to them.
If it has no appeal, it's not going to get chosen. And so in whichever area, it needs to kind of have an appeal. And so looking at the different ways and you can understand the appeal for them.
But then also you have skill set because you might like something very much. But if you're not there yet to comprehend it because you need to read it or if you don't have the background to understand it depending on the topic that it is.
So then you have what they want or what the primary receiver of the recommendation, in this case, let's say a child would want. And so what they want and their skill set.
But then it becomes a question of what they need. And that would change depending on what is it that you're recommending. Is it for leisure? Is it for learning? And so then it becomes what they need.
Then you have if they're younger children, particularly what the parents or their family background or like the cultural background says that is appropriate for them because they might like something.
And in some parts of the world, it might be super appropriate. And in others, that part of the world for that age, not so much.
And then you have what the teachers, let's say if it's for learning, would also think it's what they want. And then you have the owner of the recommender that they're also going to have some interest in.
And that's kind of the way it works and it should work. You have someone that is kind of the sometimes it's commercial. And even if it's not, there is always another factor that needs to be prioritized.
You have all these resources that don't come out of thin air. Somebody is paying for that content or developing that content. And then it needs to be showcased.
And so then you have all these different dynamics that need to go into this big recipe. And so how do you take that into account?
And so I think that in terms of this particular area is looking at the users themselves, the second that it kind of stakeholders that are going to also impact what should be the output.
And then whoever is driving the recommendation kind of items, but they're also going to have a goal.
And how do you put all that together so that everybody's as happy as they can be and as satisfied that they can be with the caveat that the harms and harms maybe it's too strong of a word.
I don't think so, but maybe it is. That is something that cannot be compromised. And I think that's the main difference that with a grown up.
I don't like a recommendation. Like I said, I skip it. Or I don't engage with that recommender anymore. Or I don't go back to that side.
As a grown up, I can make that decision and I'm not going to be scarred for life. Or at least I haven't encountered anything that it was like so dramatic that it was like worst case, like, no, this is really not my vibe.
And if I get not my vibe enough, I stop going to that platform. And if I continue to get the things that I like, we continue to go back to these platforms.
I love, for example, that Spotify picked up that I listened to music in English and Spanish. So now I can continue to have that vibe.
It wasn't always the case. And so, but with children, if we go back to them, if they're exposed to something really inappropriate to something that, for example, is violent, because we're thinking we always go to like the extreme.
But what about is something that is violent? And that's not within their culture or their family environment. That's not what they have been exposed to in any and the families don't want that.
They can really take it the wrong way. And it's more than just, oh, I don't like it or this is not for me.
Seeing disturbing images. We were looking for examples when we were recommending material for learning in the classroom and there are things and topics and videos and blurs or news that you can show.
And there are others that maybe someone older, you will be perfectly fine because it's educational content, things about history.
There are videos or recounts about history and war that when you're more of a teenage, you can deal with that because you're prepared.
But when you're younger, that's not what you want a young child to see because they're just not prepared to process it.
And so I think in those cases, the harms, there's something that cannot be negotiable.
You can also not become as a recommender. And sometimes I talk about it.
I put me as the recommender because we're like the designers of this, whether we want to or not, we're making decisions all the time when we're deploying and designing these recommenders.
And so we might not mean to, but not making a decision is also making a decision. And so if we're not very mindful about particularly with the audience can be impacted by that.
I think it's a challenge. And so not really being able to say, this can have a detriment and it can have a lasting impact.
It's beyond. It's not relevant for me. And so those that are harms that are non-negotiable.
But at the same time, that's kind of what I was saying before I kind of derailing the harm being non-negotiable is we also don't want to be a sensor machine because that's also not the point.
Trying to say not nothing is for children. Yes, there are things that I agree with that.
There are extreme cases, but we live in another with the string cases are OK.
I know that you're not going to expose children to porn like with the search engine, you have the same search.
That's one extreme. But what about everything else? The degrees of the vocabulary in the news that could be, I don't know, the bullying, the different concepts that they may be young to comprehend or that they might not be prepared.
How do we make sure as recommender designers that we are aware of that, but we're not becoming a sensor machine because that's also not the answer.
And if it becomes too much of a sensor machine, then we are somehow undermining the appeal aspect because in the end, all the people or all the children stop using them, which they might be supported by.
Then there's also if we are modeling this sensor machine, who are we modeling?
Because I see that a lot because I grew up in Argentina, then I moved to the States and now I'm in the Netherlands.
There's things, vocabulary and terminology that because of the culture in the US would be a very no no for a young kid.
Argentina is like, OK, you grew up hearing that. And I'm not saying it's right or wrong. It's just that if you grew up hearing that and it's not a harm.
So then why would I say, no, nobody can see it. And that's kind of my point.
Then if we become this sensor machine, then are we going to be the extreme and say nothing?
And then it defeats the purpose of giving that sense of autonomy and exposure so that over time these kids are developing their days and they're developing opinions and they're forming their vision of the world.
And so being able to give enough when they can take it, but not just say, no, everything is like roses there and there is nothing wrong.
That's also not the reality of life. And so yeah, yeah, deprive them of the ability to somehow explore what is right and what is not.
So at the end of the day, the recommender is not the parent. That's not the role of the recommender.
And so the recommender should give enough of the resources in whichever context so that they can take advantage of it.
But it shouldn't be the decision maker. And again, it goes back to that, trying to understand what is this recommender working and what is the society there?
And what is the background? It's something as silly as in the modeling. We were talking before on the complexity.
OK, if you're a child, you speak two languages, why would the recommender only give you one language? It's something as tiny as that.
Is it saying that one of the languages is less important? Is it making you believe that you shouldn't? Is it giving you a different signal? That's kind of my point.
And so we want to it's complex and we want to cater to the good things because the recommender can. That's the beauty of it.
But in this era of artificial intelligence, we can do a lot more. So we should be able as developers and designers to really be able to pick up on these user signals and cater to those.
It's not that challenging. It's not that the content is not there anymore or that the content is like so tiny or only in one language.
It's there. We just need to make sure that the recommender is that conduit to really tell. And so these are little things that the recommender can do preventing the harms.
And so it might not be super personalized, but it could be the little things that we can be doing over time.
Great. You mentioned one sentence that I found pretty interesting, which alludes to another important group of stakeholders, which are the parents.
Since you said the recommender is not the parents. And I guess there is a pretty complex notion of how parents might perceive recommenders that provide recommendations for children.
Since and I guess there are many or other adults, grown up people listening to this podcast and also many of them might be parents themselves and researching or practice and recommenders doing both and having also certain opinions about how suitable they are for their own children, for children in general.
So parents and recommenders. The first thing I'm also thinking about is actually trust again.
Yes.
Since in Germany, we have that term of helicopter parents. So basically parents, they kind of try to avoid everything bad from happening to their children and thereby also somehow prevent them from making their own experiences, which sometimes can be bad, but which I and for a disclaimer, I'm not a parent.
But however, I can have an opinion, I guess, is sometimes also important to make a bad experience to somehow learn and develop.
And then there are these extremes, especially when it comes to exposure of kids to social media, where maybe some parents might have more of a less affair style and saying, okay, they need to make their experience themselves.
And on the other hand side, there are very restrictive parents, maybe these helicopter parents who somehow try to constrain and restrict everything and keep control of everything, which might also be somehow not really manageable to a full extent.
So my question is, how do we build trust in a recommender systems with respect to parents because they are also finally the people who are in charge of the material or the access to those systems for their children.
So how can we make sure that we as researchers, as practitioners do the right job in building trust with parents so that children can have access to systems that support their development?
That is something that I'm still trying to grasp and understand what would be a practical solution for it.
Because again, unlike the more tested type of algorithms, when you have a lot of offline things going on that, then you can give some assurances of how things are going to work.
With this one, you have less. And so you need to be pretty certain. So I cannot design something and promise parents that 100% of the time or 95% of the time, everything will be fantastic because that 5%, if it's kind of a little bit off, maybe not so problematic, but if it's really off, then it has bigger consequences.
And so that one is something that I'm still struggling to grasp. I think that for me, the starting point is that idea of if that's kind of like the frame of thinking is the idea of putting the expert in the loop.
And so maybe perhaps the recommendations are for these expert hands first, that then they can be passed along to the children.
And I could imagine that one for like a younger audience. I don't imagine a teenager having that. And what do I know? Again, I'm like you, I'm not a parent. And so I read a lot and I talk to a lot of people and I see all the different perspectives.
But I don't think if a teenager would say, okay, mommy, give me the subset of the recommendation. Maybe not. But particularly for the younger ones. And that's something that we talked a lot with teachers.
If the recommendation can come from them and say for this classroom, these kind of level of readability that is not just one and these different type of topics, then could be the teacher, the one that then narrow it down to give more of a curated list for the summer reading or knowing what they know about their users and their background.
Because the teacher may have more of an insight that maybe the child didn't feel out of this user profile at the beginning, but the teacher will have that knowledge. And so maybe presenting it to them. The same with the parents.
If we can recommend something to the parents, knowing that it's going to be closer to what the child might like. And then the parent might make the final subset of curation knowing that it fits with their perspective.
That might be a starting point. And I would imagine if enough time goes by that the parents said, okay, every time you're giving me things that are like safe and suitable and they keep advancing and then maybe eventually they'll say child use this recommender.
And so, but this is if we think of more of a recommended environment, like dedicated recommended environment. Now, if this is kind of like something that goes into the bigger media side.
I always keep thinking of these bridges that could be built. And so something that says, if you can, if you're going to use YouTube, maybe there could be something that if something in the recommender loop of the YouTube, it's going to be not the writing that it could be triggered.
Maybe not. And so trying to have that level of help. Maybe I don't want to develop the next YouTube recommended because I'm not at YouTube. But could I be that triggered that they're missing something?
Because they're not meant to catch and it's impossible to catch everything. But if these little bridges that then the parent can say, okay, I can see how this is working and helping out.
But yes, I think that side of we go back to this very complex, multi-step group, like making the children happy, but then making sure that whoever is the grown up in their life that acts as a gatekeeper, it's also happy.
It's a struggle. That one is also a struggle. But like I said, starting with them first and then letting them experience could be a good place to start.
But that's not something that I have done in depth yet beyond interactions with some teachers.
Okay. Okay. There's another point that I would like to briefly discuss before I would like also to hover over to the community around that topic and where people can find more information.
So maybe as a missing piece there is actually the evaluation in some of your papers. You have also mentioned as one of the main distinctive characteristics between adults and children, the willingness to provide reviews or explicit feedback for recommendations.
Now we actually know as practitioners and researchers that explicit feedback is having a minor importance compared to implicit feedback.
But how do we actually leverage explicit feedback if present and especially implicit feedback to judge the appropriateness?
And you mentioned all these sets like appeal, the skill sets, the need, the appropriateness of recommendations. So how can we actually evaluate whether we are providing the right or good recommendations?
That one is the utopia, I think, of the recommender in general because that isn't yet even in general. And Metrica says, oh, it's going to capture all of these things.
But I think that with children or again, these non-traditional populations even more, there is, I want to say two years ago, a paper because I've been vocal about the different needs and different ways to take a look at this.
But there is a really nice paper on the perspective sports shop from Emilia Gomez and her group from, I want to say two years ago.
And it's a really nice kind of overview of different perspectives and what has been done and all the open challenges on how to evaluate recommendations for children.
And I think there I really like that kind of summary because there are different perspectives. On the one hand, you have some things about, let's say, the skill set.
If you can identify the skill set, then you can measure how far apart you are. Like, for me, a good measure. And I always go back to the readability because they cannot read it, they cannot understand it.
And so then it doesn't matter how much they like it. And so that one, if you at least have the starting point of, you know, at least even one thing they have read, you can see what things that you're recommending are matching within that range.
And so we already know that if kids read way below their readability, they're kind of detached and they don't really care. It's too hard. It becomes too challenging.
And so it needs to be in that sweet spot between their area and maybe a little bit challenging, but not so much. OK, now we can measure. So that's one perspective.
Then you can measure the content-wise. Well, if you know of something that they have read or the topics that are of interest, if it's not personalized to that user group.
And so you need to measure in different perspectives and then somehow put that together. And this cannot be that one is very high and the other one is very low. And we still call it good in an average.
This, I think, is the other thing. It needs to be consistent in the perspectives and some are going to be more meaningful than others. This idea of like, is it suitable?
And if we measure, let's say suitable in this idea of it's not going to be disturbing. That one is pretty important. It's not negotiable.
The readability, it needs to be in the ballpark. But if it's a little bit low and a little bit high, that one you can be more tolerant.
The content, the same as with the adult. Yes, we want to be in the ballpark, but also if we're close enough or if it's like serendipitous that you don't even expect it.
But it kind of fits. And so I think it cannot be a one size area. It needs to be that you look at different things and then there is no way around it.
You need to look at this from the user side. Are they really going to value this at all? And that one, I think we're not going to get away.
That's when we can see, at least you can say, well, they tell you, I read this and I read this and I like them both. Well, at least you have one signal and you can say, would I be able to predict this one that they like?
And that will give you an idea if you're going on the right kind of way. But there is not one perspective. And I think they are all pretty important and they need to kind of be there.
Which means that there is still a lot of research potential and also potential for experiments and practicing. But at least we do have a good starting point.
But I think that that's why kind of what I what I kind of started saying at the beginning, that's for me, it's not that I always meant to make the bulk of my career that you mentioned the emphasis.
It's not that I mentioned the bulk to be children. It didn't set out for me to be like that. That was just kind of what I ended the last kind of beat on my PhD.
I have looked at all the other user groups or experts on this and that. And that was a way to start. But then for me, what as a researcher made me very curious.
It was like, OK, what happens when you don't have the data set, you don't have 100 other researchers working on this, they can give you their point of view when you don't have ground truth, when you cannot run this.
I don't have crowdsourcing platform for children. I guess that let's say I'm going to recruit. So when all of that goes away. Yeah.
And some of the metrics you can use and some of the algorithms you can use. But when most of that goes away, how do you start building again?
And I think that's what keeps me very curious because every time I advance one thing, then 20 other pop up and say like, OK, this is now we need to know more.
And that's why everyone something that I learned here, sometimes other colleagues bring. I work with some of my collaborators on trying to help out with recommenders for people affected by ASD, kind of like autism disorder.
And so I kind of brought my understanding of we don't have data, we don't have this user. But then that one was a completely different user group that didn't have a different skill set that needed to be modeled.
And so the same with people affected by mental health disorder. And so there is a lot of these user groups that are very interested, but they're so rich to explore so that then we can better start them all.
And I'm not saying let's throw away everything we know. No, let's keep what we know. But let's also understand that there are limitations on what we can use or the impact and then build from there.
And so this idea of trying to build the community around this, I think has a lot of value, even just to have the conversation. Definitely.
Yeah. And as you said, you don't need to throw away, which has been brought up by research that has due to the data or due to the interest more, let's say, support of evidence.
So you don't need to throw it away, but you need to put it into a different context and basically question it or take it as a starting point, but then see how it fits to your problem or to the problem of catering to the needs of non-traditional.
No, I think you phrase it based. It's like it's really put it to the test and probably from a different point of view and say, okay, well, we can use. We will because this is something if there is something that is kind of accepted in the community.
And that means it has been around and it has been read and critiqued by like your entire RecSys community. Then I want to take advantage of that.
Of course. There is no point in reinventing the wheel, but really proven it from the different perspectives and then say, okay, what can we keep and what we need to kind of use it to support.
And so not just ignore it, but support it. And I think that's where the community comes in. And that's what is the value. Some people say, oh, no other people are working on this is better. I disagree.
It's like, I don't want to be that it's me and my students and then two other groups and then we just talk to each other because I don't want my vision of this.
I want the community's vision of this and the more you can talk about it and understand it from the different user groups, not just children and have these debates. It's what makes it more rich. It's what has made RecSys grow in the past what 15 years so much and made it what it is today that now it's like everywhere.
Even when you don't pay attention to it, it's there. It's writing the things that pop up. And so, but I think a lot of the growth came from that it's supporting all this platform, but it's also discussed and discuss.
I think RecSys is one of the few conferences as a conference that I go that has both searchers and the industry practitioners and they're having these debates back and forwards and learning from each other.
And that's what makes these communities so rich and really capable of evolving because they're not silos. And I would like the non-treational user groups to be more in the scene there, not just the one paper in the proceedings one year and then the other paper or two in the proceedings another year.
And you are doing a great job in that regard about building the community because it was, I guess, RecSys 2017 where you have been co-organizing the first Kit Rec workshop, which I guess has been followed by many others.
And now we have seen the fixed Kit Rec workshop, not with the RecSys, but with another conference. So can you share some details about where people who are interested in diving deeper into these topics, find more information, can connect to fellow researchers and practitioners.
So where can we go and see you and all the others and discuss these topics?
So that one, the Kit Rec one was like, it's very weird and near to my heart because it kind of took a while to put it together and RecSys was very welcoming that first year.
But we also noticed right then and there that there weren't a lot of researchers to keep it going at RecSys at that moment. I think that now would be a good moment to bring it back and not just at Kit Rec, but now with all the interesting on the non-treational user groups, extend it a little bit more.
That first year at RecSys we understood that there was still much more to learn about the trial as the user of a recommender system. And so over time we kind of realized that having it in a conference like an interaction design and children, we could bring more multi-disciplinary perspective and bring the human computer interaction community and the child interaction community, but also teachers. And so we have been holding that every year, but it's always good to see that there is always one or two RecSys people that make the trek to Kit Rec.
With Kit Rec we also extended it not just to, we call it Kit Rec, but it's not just recommender, we extended to search because again, we wanted to understand this user group and these limitations and the ethical considerations that they would apply to both.
And so we continue to kind of build community there every year. The other thing that I started to see that for me was great. The last two years at IUI and at Yuma, there were workshops socialized and happy that they were explicitly calling for focus on non-traditional users.
And so that was another way to bring, they were not limited to RecSys, but it was another way to bring the few colleagues into the discussion there and meet other people that were working on non-traditional user groups.
And so that's a way to keep it going. And sometimes being like a little bit, not loud is not the word, I'm loud in tone, but not in the topic.
But really continue to kind of beat the plaque in the year. Sometimes I give chats about recommenders in general or my information with your work and I always make a point of bringing back this idea of the children, non-traditional users, because I think it's important that we keep, because sometimes people don't make the connection.
And then there's like, okay, another type of user. And then you mention a little bit of like, but the evaluation, and then you get that moment and then people have opinions.
And I want those opinions to be boys from people that have a lot of experience on this. They just haven't applied to this, they can apply to something else.
So we continue to have these chats, but I think what you to have a nice revisiting, maybe a workshop and trying to bring back the different perspectives on non-traditional user groups and how that impacts RecSys.
It was great to see the last couple of years in perspectives how a lot of evaluation on non-traditional user groups have that not just in general, but for them. And so there's always something that keeps it going.
And I think that RecSys is a great track also another of the workshops that at some point there's always a paper or two. And so participating there, it's always interesting and it's good to see it going.
But I think what you I'm going to start bugging some people, not for this RecSys that is coming up soon and everything is already settled.
But I think that it's time now after, yeah, from 2017 till now, I think it was you to come back and revisit that at RecSys.
Yeah, and I guess there was also some contribution in 2019 for the complex rec workshop. So the paper about the seven layers of complexity of recommender systems for children in educational context.
Yeah, that one when we read the call about the complexity, I was like, we know a user group that is complex. And so that one was good because I got to work with a teacher and I got to work with a researcher in human computer interaction and a researcher from industry.
And so then we kind of have this paper there. It was kind of this discussion of saying, let's look at all the perspectives and the moment as well.
It's the evaluation. The person from industry was like, but yes, somebody needs to get the commercial aspect of this. Like, okay, another perspective. Oh, but the ethics, the horrors of this because they're very challenging like ethics and fairness and bias concepts that are really challenging to begin with.
And now we have another layer. And so that one was a good reflection. Oh, yeah, that I had not good of like, Oh, look how I like my paper, but it was a good moment to sit down and really good discussions.
Yeah.
So a very interdisciplinary research work and definitely worth reading the paper in 2019. And as always, we will try our best to include all the papers that we alluded to in the session in the show notes. So find them there.
Wow, that was really intense, but great and broad talk about this important topic. So, so I really, really appreciate it.
Well, no, thank you for listening to all my thoughts about recommenders from non-trivial users, particularly children. I'm like, very dear and dear to my heart and I continue to be very passionate and curious about it.
But yes, that is not everyone's cup of tea. So I appreciate you listening to all these long discussions about this.
And if people want to reach out to you and discuss more, will you be at RecSys or where can people meet you or reach out to you?
This year, I don't know if I'm going to be at Rexon. I'm already crying about it. It's like projects and gatherings that are all co-existing at the same time.
Yes. And so I think I'm like, I'm going to have to miss it and I'm going to cry about it and then just have FOMO and vicariously live through the WhatsApp that people send me and the social media.
But if I'm not there, I'm always happy to respond to an email. And I made a point since it was the summer to updating my website with the contact information and everything.
Yes, I did my homework. So I will be always I'm always very happy to chat about the side of my research. So I always welcome an email.
Great. Yeah, I can fully support this. So I haven't had to convince you to join the show. So that's that.
That's actually very great because it's always nice to have a new perspective and an expert like you in the show who can talk about these things and provide pointers and share experiences.
So this is very, very worth and I hope that the listeners will seize it as well. Actually, it wasn't like that.
I haven't heard your name before. Michael X trend recommended you in the show. But the recommendation was somehow for me like, okay, but now I should finally reach out to her, which I have somehow on my on my bucket list of people that I want to reach out.
But actually, to keep with that theme, is there a person that you want to recommend for having in this show?
Those are like good questions. So my the first name that comes to mind and I will find you a paper so that you know and some of the works that I have.
I think that's because if not, Federica Senna, Federica and when we have been working quite a bit on recommenders for people affected by autism or people in the autism spectrum.
And I think that it has been such a great work what they have done and quote of their work. You're going to see it on the user model inside of it, but all apply to recommend there.
So both of them have been doing fantastic work. It's worth discussing.
Let's see. I'm a fan of all the recommender people. Well, of course, Peter Brusselowski and he's doing a lot for education. And so he's also always a very fun person to chat with.
Oh, yeah. I'm sure you know. Let's see who else. I don't think I've seen on your list. Kristoff Trappner.
He's also now working a lot on the news, but he also did a lot on the recipe side and the health side. And again, it's a completely different take on.
There is no the relevance concept or there is not a lot of things. And so I like that quite a bit. Marco Talsec doing personalization.
His work, I really enjoy tremendously because he has a depth of understanding the user modeling side from very different perspectives, even if it's from the traditional user or not looking at the personality aspects and and these different user groups and the context in which the recommender can be embedded.
And it's always such thoughtful work and he always have great opinions. So he would be a very good, a really nice conversation.
So those are the people that are the top of my head. I would say they're just really nice to sit down and listen, you know. Okay. Excellent.
And learn from and so those are the people that I would share your way.
Cool. So, so plenty of people to reach out and to feature in the show and talk about their expertise and their topics and of research or practice and recommender systems. Great.
Cool. So, Solé, thank you very, very much for participating in this and for sharing your thoughts with the community.
I hope they were somewhat decent. So they were very real. But like I told you, you never know what you're going to get.
So I thought that they were very honest thought. I hope they were like useful thoughts to a degree. Some of them were great. I definitely believe they are useful. Some of them at least.
But yeah, that's what I hope.
People want to go even more deeper into all of these topics. Then, as I said, go to the show notes and then you find all the papers.
And from there you can go further and further and then have deep discussions with Solé.
I'm always happy about that. I am always assume away or the WhatsApp away. So cool.
Then again, thanks for participating and have a wonderful day.
No, thank you. And good job. Good luck with your new job.
Thank you. Bye.
Bye.
Thank you so much for listening to this episode of RECSPERTS, recommender systems experts, the podcast that brings you the experts in recommender systems.
If you enjoy this podcast, please subscribe to it on your favorite podcast player and please share it with anybody you think might benefit from it.
If you have questions, a recommendation for an interesting expert you want to have in my show or any other suggestions, drop me a message on Twitter or send me an email to Marcel at RECSPERTS.com.
Thank you again for listening and sharing and make sure not to miss the next episode because people who listen to this also listen to the next episode. Goodbye.
Thank you.