foreign next up we have Cancun Wong who has been an R convert since late 2019 when a simple question turned them towards the language we know and love and all of us kind of are familiar with the question how can we make sense of all these numbers he is a proponent of experiential learning and in my opinion he is also the person with the sweetest bio in 2022 he experienced probability Theory by dedicating to uh by dedicating 24 bookits to his wife which is estimated to be about a 6.6 percent per
chance per day to his surprise there were several occurrences of back-to-back purchase of flowers and that was a great example of experiential learning which he shared with us Ken will take us through a fascinating journey of our shiny dashboards and how we can use them to rank candidates effectively making the recruitment process diverse Equitable and inclusive so I'll hand over the stage to Ken and here we go hi hi Divas thank you for for the generous uh uh generous introduction here I'm just
you know thinking how does one follow Eric's presentation uh um but we'll try um it I'm happy to be here I learned so much uh with this shiny conference um I'm just gonna put it out there uh we're not using modules yet but we will after this I'm here to share with you an educator a medical Educator's Journey uh to data science um one of the the examples that we'll be presenting or demonstrating today is residency applicants ranking dashboard and algorithm um and I was I'll explain a little bit o
f what that entails um I have no Financial disclosure and data I use here are fabricated or semi-fabricated to preserve some of the interesting patterns um and there are weights that are used here uh again uh I reinforce that that's not it does not reflect our prior or current uh ranking values so who am I I am an infectious disease physician by trade I'm also an associate program director for Internal Medicine Residency program at Cleveland Clinic Akron General I call myself a data science Hobb
ies I love our ever since uh uh it became a convert or is this just so heartwarming the r committee Community the shiny Community um guided us um and also showed us how how to you know solve our technical issues and also how to answer the important questions so till thank you this is a map of United States uh a highlight in red here is Ohio so for any of you um you know who wants to come and visit us you know we'll take you around now here's the interesting question that we had a few years ago u
h imagine yourself um you have 174 recruit candidates now how can you rank them meaningfully after an interview so just imagine yourself perhaps you're a CEO Epsilon or a pretend uh a make believe CEO of some companies your startup company how does one um you know use this information and rank it more meaningfully typically anything would start with the workflow of conduct the interview gather data rank them and then you know wait for match or or offer in the real life setting you would you know
once you give an offer a person would accept or not and those kind of things but in a in a a residency uh matching is a little bit different so I'll explain a little bit of what that means here in the United States before a person a learner becomes a an attending physician a we have to go through a residency program which is a training program um and it's not so simple or straightforward where um you you will go for an interview and then you get an offer sometimes it may it can be but majority
of the time it has to go through something called matching or um ranking so for example here we've got candidate a who had gone through one through n interviews and here you can see on the red uh it's Us Akron General let's say candidate A ranked this as number two and then we as a program would have to rank our candidates or interviews uh interviewees as well um through through the process um there there is an option for both the candidate and the program to not rank a program or candidate so t
hat is an option so once after this rank list is created uh at the end of January all these get submitted to into this um what we call nrmp which is national ranking matching rank matching program and they have this algorithm that that attempts to match um the program and the candidate and in March so this week um they will release the result to the candidate and also to the program and the candidates will know whether they match to a program or not and the program will know whether all their sp
ots are filled so um in our example we're going to focus on the program's perspective of ranking candidates um so since this week is match week if any audience had had gone through this process um and and you know had successfully matched I congratulate you for doing that but for people who did not match don't give up keep at it add more skills soft skill hard skill technical skill to your to your Arsenal and then keep trying don't give up is the key this is the evaluation evaluation sheet that
we have um each year we try to revamp it a little bit different but uh just just to give give you some perspective um so we've got the features or variables on the left side and then it's a very simple likert scale one through six um with some criterions anchored to the score itself the challenges that we had um is you have 174 or perhaps even more of those information how do you how does one interpret those data it might sound simple for most of you data scientists but for us Physicians so we w
e have a hard time uh trying to to look at this number you know through our own heuristic Vision I mean um and other questions that we had also too is um is there a difference between International graduates and local graduates so just just to give you some idea um the The Residency program doesn't only take local grads from the United States but they also give an opportunity for international grads who had trained in other countries so we're wondering is there a difference between those two gro
ups in terms of interview scores and then how do we take all those into perspective and and rank them um meaningfully not only that you have an algorithm you have a ranking how do you push that into uh to to the to the leadership world uh where they can use this tool and and be informed of you know which which person to rank and so on and so forth and lastly we're we did not know how to code uh uh you know what is r uh all these are very very foreign to us um and and which is which is very excit
ing this was our prototype one don't laugh um it's uh it's it's a spreadsheet full of information of course you know uh confidentially stuff is being blocked um it worked for a little bit until it didn't um so we Embark into the Journey of to try to learn data science and try to learn coding you know you've got we've got problems with numbers um but we're so lucky and so fortunate to have these open source Community uh especially in R um and uh you're just able to learn things uh through free re
sources and sometimes paid um packages like tidy verse and then you add on shiny tour to to it and and with all this dashboard and then learn so much technical skills along the way to be able to make a dashboard where it's almost like just uh just like a window into into uh into the you know the data so that you don't have to manipulate a lot of the dashboard um uh from from uh from a server perspective um and and just you know the data is the one that you can just change things and and use a UI
from there and through this journey I have a newfound respect for data scientists and data engineer it's a lot of work uh it's uh so so I I truly understand the the the the challenges behind you know it's not just it's not just learning new skills and applying it but it's failure after failure and debug um it's that's that truly um uh put data science into this perseverance perspective so I I that's definitely something that I've learned from from learning this so this is our um first page for
for our shiny dashboard um uh on the side here we have our site panel um and this what we're looking at is our first page of summary so I will put just the evaluation sheet on the top right what we're looking at here is um we have a question of the overall impression of uh of of of this candidate how do you where would you rate this uh candidate um and so to answer the first question of interpreting data we never really thought about using visualization until until when we started learning thing
s it's it's just looking into just putting in the perspective where this candidate is is a very very top compared to most of our candidates and you can see as a whole um where the other candidates are are from from the plot um so that's that's very interesting uh Insight that we've discovered yet so simple uh they were never thought of um and because this graph was made through plotly um uh we're able to just you know hover it able to you know quickly see which candidate is next to this person a
nd whatnot and you know we use DT for uh uh uh data data frame I'm sorry table and you can see here there's very little uh inter Raider variability which is a good thing and not only that each evaluation should come with some comments whether a person is scoring this person's High marks or low marks issue come with the comments and that really at depth into how this how how this candidate is and and we'll have different section on on the first page and it pulls from different areas as well next
is a little bit more in depth um into into all the features and you can see how this candidate fare compared to the rest of the candidates from the maturity standpoint we do utilize lancioni's ideal team player framework we do ask behavioral questions and probing questions as part of the process to get a little bit more details as to you know whether this candidate is hungry for something or is this uh are they uh Smart in terms of tackling problem then the next question that we had was is there
a difference between International grads and local grads um you know is it fair to compare um all of them together or should we split them up um is there a random effect that's in these two groups um so just give you into perspective um on the on the right side of the plot is these are local candidates and then on the left side these are internet International Medical graduate um they're at least according to this data set there may be some uh uneven distribution distribution of the overall sco
res um so it either means that uh the local grads the scores of local rest is a little bit higher because they are they are they are very good or our interviewers are ranking them higher um so we can explore that a little bit uh further down the road um the other question that we also had was is there any communication difference because for a lot of our inter International Medical grads um uh because they English is not their first language um so you know is there a a difference in score compar
ed to local um and at least from this data set there isn't any and with all these uh taken the place how do we use all this information and and put it into a formula this is when multi-criteria decision analysis uh comes into play um it's it's a very simple weighted sum model um so each variable has a weight um in front and and it depends on the programs um uh uh decision in in terms of would like to increase a particular uh attribute uh uh in the cohort and you would increase the weight a littl
e bit so I know this is a zoomed in version of the of the rank uh dashboard but the bearer with me that is very busy on the left here uh these are our variables or features again all these are very uniformly allocated for demonstration purposes but one pro the program could Inc could increase um a verb of the variables or weights such as in the DI or uh or you know try to increase our female Learners so these are the things that we could do to to try to you know hopefully decrease bias or minimi
ze bias or neutralize bias in that sense and try to increase the diversity um in in our learning environment once all those weights are put in um you know using a normalized um score for from all the other variables uh and you know you would get a final rank score and and then of course um inversely you would turn it into a rank so that's how that's a very simple Model A simple algorithm that we can do years or years over years and we can customize it using shiny um and add on a little bit more
in the future or take things out if we think there's you know there's no need for this maybe there's a correlation this between these two variable um so now there's also a very interesting uh question and uh that we have with this is um you may not know that in the United States medical schools there are two different types and there's one that is traditional that with a degree of medical doctor and then another one is doctor of osteopathy and their licensing exams um have different range scores
the maximum for for a do is I think it's like 700 something we I actually don't know what the max is um and same thing with the medical doctor as well because the max score is actually not publicized um and and the the interesting thing is like how can you how can you use those information when it's like an even to to actually you know put it into one variable um and uh so that was very interesting for us to try to tackle and create and you know make it uh as Equitable as possible uh the next t
he next uh page is a work in progress but I really enjoy looking at this it's not just looking at our candidates but we're looking at our evaluators how they're evaluating um is there a is there a system issue is there you know are we are we rating things uh accurately so again these are these are fabricated uh initials um but looking at through this three evaluators uh or interviewers um you see um perhaps a little bit more skewed on this setting so this is a turquoise color so it's local candi
dates so this interviewer tend to perhaps rank the local candidates a little bit more compared to this and this um evaluator so and that opens up a lot of questions the question is are are these two evaluators uh not assessing correctly or accurately or is this evaluator uh highest or is it our evaluation form issue the criteria is not explicit enough so it opens up a lot of questions but those are good questions to ask to so that we can tune or evaluation and how do we share this data and algor
ithm among the the uh program leadership this is when shiny shines um very simple we're we've got a private data on a Google sheet and Google Drive and through Jenny Bryan's Google sheet for API with an API key and here comes shiny um and with all these other packages to make things more interactive easier to find easier to locate with DT packages um so so it's you know it's it's a very simple shiny app really but it's uh it's it's so helpful for us the limitation um for our project here there's
one of the many limitations that we had but the first thing that we're trying to correct and fix and investigate is more of an inter-rader score variability um how can we because you know the interviews interviewers um could could score things a little bit differently and how can we minimize that variation sometimes we do want some variation because perhaps it might capture some signal and optimal algorithm is also interesting question as well um because the target is very different every year
um there there may be a year that we have to increase a certain amount of attributes in in our learner's environment or another year will be another um so this is uh to try to fix everything or adjust everything I think that that might be a little bit more challenging but it's I think it's doable um in conclusion shiny dashboard really really um uh save us time [Music] in the past we used to have a lot of um a lot of meetings in between interviews uh to try to to create a rank score but now we m
eet perhaps two to three times during the interview season and two the the two times that we meet is is before the interview uh we actually sat down and talked about what are the weights uh what are the uh the attributes that we would like this year um beforehand and correct our evaluation sheet and make sure things are um uh succinct um and then the third one the the one and only one would be um when we have all the data set um and that's when we look at the algorithm and then we rank um and we
this is a tool for us um the the program directors do have you know clinical judgment and intuition to towards who who he or she thinks will be best before the program and then using this algorithm and tools to help to fine-tune the final rank lists and we do think that um this weighted algorithm is able to help us to to perhaps minimize or neutralize bias in that sense um because because we're actually using all of the information that almost all of the information that we have um to create th
e final uh final um uh result um I really enjoy this piece where we have continued Improvement and the opportunity for self-evaluation um because that's how we can improve and that's how we can um uh look and see what what is it that we want to tackle this year um to make the interview process much much better and of course more ideas uh the more we do this the the more we think oh hey maybe we can apply this to that and and Link this to another thing and it's just it creates a lot of creativity
and innovation in the sense knowing the the fundam uh the foundation fundamental behind uh uh uh uh how to create this um finally this project is it it's it's not a continued a success without a team um The Physician leaders Dr The Walker Dr Solomon um they've been very um uh very engaged and collaborative in making this happen uh it does take an Innovative leader to to you know give give uh freedom to you know to your to your team to try to create something something new something something ha
s not been done before and try to you know stir the pot um so so so thank you for that opportunity for us to implement this and Jennifer Hayes is our program Direct uh program uh coordinator and she's uh she's she's she's put so much effort into into making this happen from uh uh from a logistic perspective you know making the interview happen making it smooth making sure the data is there and of um and we also have you know local uh Community uh uh Cleveland our user group um you know if there'
s any questions technical questions that we have with certain things you know we can always ask them and they guide us to the right direction or sometimes answer the question um and and also shubayu for helping us out with you know far uh fine-tuning the algorithm too um if you've got ideas you would like to collaborate you know please feel free to contact me um it doesn't have to be medical um you know data science and R and all those things it's just so it's just so interesting I'm trying to s
olve a problem is is I think that itself is is uh that's that's what really Sparks the the the the interest um if you've got feedback on this um please uh send it my way um uh thank you fantastic thank you so much for this amazing uh presentation and sharing with us this amazing tool and you know while I was watching your presentation especially the open source slide it was incredible to see you know how it really echoed what Philip shared in his uh keynote uh a day before that we always build o
n top of each other and I think that's really one of the key things that I've seen Echo throughout this conference that's so much open source contribution leads to these awesome Solutions coming up and I'm really glad that it was shiny to you know and build this and we have a few questions from the audience so from Stephen we have did you compare how the score of Prior applicants might be a predictor of Resident quality by any potential metric of how well they do um no we do not uh and did not a
nd and there's a very very good question it is it is something that is uh definitely an opportunity for us to do um it can be a little bit challenging for us because the again the the features and the metrics are always changing and we even change our likert skill uh uh every year um we sometimes do six something to do five we try to create a a cognitive dissonance so that our interviewers are not just straight lining things um but that is a very good question and um we're still trying to figure
out how to do that when we have all these noisy information great great and uh from Timothy we have this question about you know the ranking committee itself assuming that there is a ranking committee how difficult was it to you know sell this solution to them bring them in from the Excel sheet that you showed earlier to something algorithm slash dashboard base what was it a major change in how the matching was being you know done and what challenges you face may be possibly you could shed some
light on that oh yeah absolutely uh that's a great question because it does have to come from top down um you you have to have a buy-in from from the stakeholders and leaders and this is when trying to tell a story and create a motivation behind it and that's that's the I believe that's the key persistence is a key too I think it did the first year when we did this um I think most of us know it's there uh but it didn't take on it didn't really take up until two or three years later um so it's u
h I I I think if you believe in something and and and be persistent about it and just you know get some evidence behind on on you know hey these data scientists are doing this that company is doing that um and try to sell that and and and and go at it I think I think that's a key at least it worked for us it's a kind of like a proof of concept in the industry and you know that validates the approach itself uh when you're trying to sell that so I think that's that's definitely something all of us
do and we have another question from Curtis who asks what package did you use to implement mcdair also have you used applicant rankings course to delete it scores for I am um so two two-part questions the first one the mcda uh there's no package it's it's just it's just a formula um so it's it's very easy you can just put weights times variable plus you know it's a it's a it's a total sum of the model and again our our variables are normalized so you would might have to you know create a functi
on to normalize the data and the second question can you repeat the second question absolutely uh have you used applicant tracking scores to predict the ite scores for IM uh no we have not um not not from the ranking perspective we did have um other uh we have used ite uh so ite for those people who don't know what that means it's an in-service training exam we do use we have views in training exam scores uh using machine learning and random fours to predict how a person does when they graduated
from a program um when the uh and and when they take their board examination to see if there's a if they pass or fail so we do have that and we still actively use that to determine which learner from in our program requires more assistance in in test taking also I think we have one last question which ties in a couple of questions that we have uh which is about improvements further improvements except you know the model modularizing the app for example Celeste asks is the data fed into shiny ex
clusively from spreadsheets have you considered building a form for evaluators to you know input their rankings and where can you you see more improvements in the workflow itself absolutely absolutely yes we have we have thought about using a form to feed to feed that into into the uh a shiny dashboard um we do use another program which a propriety program for virtual interviews um and that itself also uh can serve as a a platform to uh to put in the scores and the evaluation um we we do uh we d
o face a challenge where the the evaluation sheet that we have uh intuitively when when you put it on paper or that PDF it it's easier to fill out and if we were to put that into Google sheet it's a little bit hard to you know you got to scroll things down so it's more of a UI experience um so so yeah if if we can if we can make it as good as what what we have is one page I think you I think we you're right we might be able to make it more uh uh autonomous awesome and so so it's big so it become
s like a design problem that you would have to solve and I think given the track record that you're actually in you know just learned shiny and mastered all that I'm pretty sure you're going to track that definitely and this is an incredible journey thank you so much for sharing this with us this was an incredible app to you uh thank you so much Ken this was just great oh thank you so much it's it's happy I'm happy to be here and and I learned so much so thank you all for for all your help and a
ll your guidance
Comments