once we've shipped our software we want it to keep working that's a given but there are two main schools of thought on how you guarantee the software is going to work once it's in production the first is get it right before you ship it that's professionalism you add tests you add types you add a QA department and you make sure it's rock solid before you press deploy the other approach very different is the Earl Lang philosophy you just say it's it's going to crash sometimes so let it crash and t
hen put your engineering effort into dealing with crashes so gracefully that nobody notices now there are a number of projects springing up these days looking at that choice and saying why not a bit of both so this week we're going to turn our attention to Riot which is the brainchild of Leandro aera and it's his attempt to bring the runtime system of erlang to the compile time system of O camel how do you do that how do you take the best ideas from erl Lang's beam and make it work with one of f
unctional programming's favorite compilers how do you make it perform and what is it about oaml 5 that makes that approach more feasible than ever I can promise you that Leandro is going to make it enormous fun to find out so let's get going I'm your host Chris Jenkins this is developer voices and today's voice is Leandro estera my guest today is Leandro estera Leandro how are you hi Chris happy to be here thanks for having me glad to have you I I gather your house sitting in Stockholm this week
correct correct I am uh cat seting for a friend actually so Leia the cat uh a little white cat with like the chromia two different colors two color eyes yeah like daid Bowie syndrome exactly kind of like a Bowie cat and might swing by here and there hope that's okay awesome yeah yeah we love guest visits by cats nice so I've got you God you're doing a lot of interesting stuff in the world of O camel and erlang and thanks yeah I mean there's there's quite a few projects that have been happening
on that space uh I've been working in both ecosystems for nearly a decade on different areas but open source stuff more recently right yeah the latest one is Riot which I think this is one of the things we'll be touching on today absolutely that's a long time to be in that space you must be an early adulter of O kind of oam definitely erlang right oh I don't know if it's in that order because I feel like llang has been much more stable for Building Systems right like for longer not to say that o
k camel isn't but for example I remember back in 2014 2013 right I was thinking I was sort of stuck in that rut of the web developer realizing for the first time everything I do is move Json in and out of a database right into a forum on a web I like oh do I really want to do that like there has to be more to this my problems maybe are not big enough right yeah and uh I taught myself lling because I thought that not having like a proper academic qualifications as a software engineer or a compute
r scientist right it would harder to get access to to bigger problems unless I taught myself the tools that are normally used to solve bigger problems right so I went through uh the learn use from irland book and I you know it was it was pretty obvious to me that this was definitely a tool that was for larger problems than I had dealt with before compared to Ruby yeah and uh I think even then there were already quite a few earing jobs for people to pick up it's still a niche like we're not talki
ng go here we're not talking like Java right but um it was already pretty well established in the sense that it had practices right around how to start a project how to grow it how to uh ship it how to maintain it like you know a like coming from from the Telecom industry where they had very strict requirements from the industry to be able to ship something far away into like you know a switch and be able to reprogram it remotely right they needed all of this sort of let's say uh Dogma around ho
w to build something right established practices let's say let's say established practi bcp's best current practices yeah yeah yeah exactly that that so I feel like erling has been more established from that angle than OK camel has that's interesting we we kind of associate erlang with the whole scalability fault tolerance thing but the idea that it's established a lot of productionizing to lie it's not the pretty kind of product like uh production did I say that word right yeah yeah yeah you sa
id it right but I think I might have made up no worries real it's not a very pretty like story to be honest it's kind of like a Engineers engineer kind of story like you have the tools you have to figure out to assemble them together but just having OTP right which used to stand for open Telecom platform which is like the big framework of philosophies and actual code right for designing erlang systems it's something that most programming languages don't have right most languages evolve there but
llang was like hey you want to use language here's a big book of how to build systems in earling yeah yeah so we're gonna get into exactly that the the fact that most languages don't have erlang and what you've been doing about that but in order to get there I think we need to start with oaml why what tempted you away from llang into o camel oh that's a good question so I think one of the reasons that I jumped ship or let's say I had one feet on each ship was that Airline even though it's super
scalable in terms of runtime performance or how many concurrent users you want to have right I think it has a problem and it's a problem that every Dynamic language has and that is the more code you have the more complexity you have right the more cognitive load your developers need to be able to juggle right so yeah as your system grows in terms of U complexity like features you have there and the interactions between them but also the amount of code you have to deal with right like different
dimensions of scalability right you need more and more tests for example to ensure that some basic things are not going to blow up in your face um it does have a lot of tools and I guess we'll talk about them at some point uh to prevent those failures from taking the entire system down right but that still doesn't mean that a single person can board into a codebase large let's say 500,000 to a million lines of barlang and comfortably make a small change knowing like confidently make that change
knowing that oh okay everything's going to be okay right you might have to run batteries of tests and tests you know they don't show the absence of bugs they just show the pressence so bux right yeah yeah yeah so from that point of view at some point when I was working at Clara as an airline programmer right um a colleague of mine Daniel McCain excellent engineer by the way he kept pestering me to learn hll you should be learning Hull it's like why what what is so good about hll right and we eve
n had a couple of sessions where we chatted about Haskell and we wrote the little basic interpreter in hcll yeah and I was like yeah I guess I can see how this works but it didn't quite clear for me and um through that conversation I started looking into other programming languages that were typed right and obviously you know you have hle you have the scalas of the world right and I found Edris which was maybe Edris yeah wow not many people go on that J yeah okay yeah especially not straight fro
m Dynamic to something like that right yeah um like I had experience with type languages such as like C before but nothing like hll or Idris so I wrote a little bit of Idris and I thought this is really cool right like the ability to have data that carries proofs right like the auto proof feature that they had was like mind-blowing to this day I don't think I fully comprehend it right but it's kind of a research language so you're allowed not to understand the whole thing because the author does
n't right exact exactly they're still researching a lot of things right and uh yeah I think the the main idea I got from that was that instead of building reliability in your systems the earling way which is having supervisor trees that make sure that the application is reliable or resilient to failures that will happen yeah the Idris approach says we're going to build an application that by construction by Design right can never fail or you know handles all of the possible errors so when when I
took that idea I was like I I love that how how can I use that what else is out there and so I found real world of camel just dialing down the type safy is something that I could understand yeah yeah Idris might be might have Fair claimed be the the peak of type systems astronautin yeah yeah I think so yeah and as you come slightly back down to the mountain to breathable air you probably do get to o yeah it's like in that Spectrum OK camel is like the go of functional programming languages and
I'm like yeah I'm want to I'm want to stay there okay so what what what happened next with your o camel Journey so I started learning o camel and I started doing some interviews in O camel and I had one interview that went very poorly because I didn't know as much of camel as I thought I did so yeah like they asked me okay let's just like build a like a transaction lock between these processes and I had to Google how to open a file I know it's embarrassing yeah but over the years I've been I've
been trying to use more and more OK camel in different places uh years later um I joined a startup and we had to build tool link internally so we build a system in airling and we had to build some tooling to be able to synchronize data essentially a model right AC across the back end and the clients and we had to do a lot of like let's say mix and matching of external data sources right into our own data model and since I had been working before that at Spotify and I have been exposed to ontolog
ies I thought we should build an antology for this right okay and we built a code generation tool in recent ml uh which is just a new syntax or I guess not so new anymore but it was a new syntax for Camel right that was more approachable syntactic sugar on top of camel right essentially it's just like what if a camel had semicolons right the most important question in programing what if we added semicolon exactly would would semicolon fix everything and turns out it kind of did for a lot of peop
le right okay yes so we build this tools there and and we were able to maintain sort of the the generation of erand code and and also JavaScript code right all from that tiny little project and and that was really nice and since then I've been on and off in different companies using either recentl or camel and more recently rescript right or what's rescript I've not heard of that one glad you asked so OK camel eventually got this new syntax was called reason but the community reason had sort of
two purposes right one of one part of the community really wanted to be closer to the web right they wanted just to bring the okel type system to the web and the other part wanted to use the the recent syntax to write native programs right and the tool chains kind of started diverging and at some point there was this hard split a whole Rebrand right where people said you know what we're going to grab whatever we have right now and we're going to start building exclusively for the web and we're g
oing to hyper optimize the language the syntax the tooling right so that adopting that language in a web project right is just chefskis right right and they did it's just a a blast of language like today if you ask me hey Leandro let's let's build a new like startup and we'll do web things I'll be like yeah we're going to use rescript for that for sure okay cool uh and that presumably just compiles to JavaScript or correct yes so there's this whole compiler that used to be called Buckle script i
t's called mange this day right oh yeah that uh it's a sort of it's a source to Source translator and it compiles to Super idiomatic JavaScript it's it's kind of scary like when people look at it and go like whoa how did it do that you know because it even includes support for like jsx syntax so you can do like your typical react style like components and everything and it translates those correctly and you go like oh holy crap this is excellent yeah okay that sounds like a lot of fun that sound
s like something we could dedicate a whole podcast to yeah yeah absolutely absolutely I I will give you directions of who to talk to to bring for that cool cool but you stayed more in the backend world right uh a little bit um so I to me like the distinction it's more about like what kind of domain problems you want to solve right and what customer problems we want to actually get into because there's a lot of customer problems you say no I don't want to solve that right as a company um so most
of my career I work with startups so i' I've gotten used to have to sort of move around and say like okay now we need you to do infra or devops or backend or data engineering or these and that in fact today I'm like my day job is being a product manager so like literally don't write any code today right no yeah exactly no I'm actually very happy I'm but um anyway so I think I did a lot of back end work because it's the place where you have the most freedom to make bad decisions right in the fron
t end is likee it in the front end there's a lot of like constraints from expectations from the rest of the web Community right like you want to use react for example it comes with a bunch of different everybody still does things their way right but there's a bunch of ideas right and Frameworks and subf Frameworks and Library that just put a lot of constraints and back a lot of expectations about the right way things should be right exactly yes yeah yeah in that sense I suppose the back end has
uh fewer stakeholders to worry about the way you're solving the problem yeah yeah it's like as long as you give me an API and the API is doing the thing then I don't really care how you do it underneath for the most part right yeah yeah okay so we're going to get into the earlness of O camel that been working on there's one more step I think we need which is you told me that this was changed significantly by the introduction of oaml 5 correct so tell me what's interesting about oaml 5 right so p
rior to OK caml 5 OK caml four and Below have been historically single threaded right if you wanted to run an application it will run on a single OS thread even though there is something called thread in no camel which is not really a thread it's just another thing please don't use that uh okel 5 introduces two things the first one is what we call domains which are actually OS threads they have a little more like I guess they call something differently because there's some more magic to them rig
ht that allows the garbage collection to work well across them and so on but basically you can say hey OS I need a thread and then you get a second thread and now you can actually run parallel code right between those two cores MH and the second thing is that it introduces effect handlers and this used to be called algebraic effects but apparently been corrected from the presentation I gave a couple of days ago about bobc they're not called that anymore now they're called effect handlers the alg
ebraic partes was dropped is that just pure marketing because the word algebraic is scary I don't think anything about o camel is pure marketing to be honest fair enough yeah okay effect effect handlers tell me what an effect Handler is all right so an effect Handler is basically a TR catch right like that's the the easiest way I can put it so everybody in every language can be like oh yeah I I grasp what they do is to say you have exceptions in most languages most of the time you can do race or
throw or something like that and then you raise that exception and as long as you have like a TR catch somewhere above that like in the here key of the program right then you'll be able to say oh I will catch that exception right and then I will do something now now the difference is that once you catch that exception you can do something and then you can say let's continue the code exactly where the exception was raised now when you do that if you you know generalize it or like change the name
from exceptions to effects now you can perform an effect anywhere in your code and as long as you have the strike CCH which we call handlers then you'll be able to intercept that effect do some work and then eventually if you want resume the code exactly where it was left so okay give me an example of that that isn't exceptions yeah absolutely so for example um I mean one of I think the example I want to give you is like how we use handlers in Riot but uh we'll get there we'll get there we'll g
et there let let's say you have um a function called random right okay and it will give you a number between zero and one right you can Implement that as an actual function or you can implement it as an effect so if you implement it as a function if you want to like replace what that function does you kind of have to have something like a mock or inject the function into your work right so you can replace it and so if you want to write a test where you want to see what happens when the random re
turns zero right yeah then you can do that now let's say we don't have a function but we have an effect so we call it perform random number right yeah now that code without a Handler doesn't do anything in fact it fails in the same way that racing an exception without a tri catch does right unhandled exception andh handled effect compile time error no oh okay no not compile time like even in camel exceptions are not typed like obviously there's a type of exceptions you can patter match on them w
ith types but they're not tracked in the signature of the functions right okay uh which is different from other languages like even Java where you say like this has checked the exceptions right or Swift I think Swift does this super nicely where you can say this function throws and then if you call that function inside of a function that doesn't throw then you get a compile error saying hey you need to tell it that this function also throws and so on yeah yeah but anyways so now we want to grab
that code that has an a Handler an effect right and we want to run it for a different a few different tests in one test we we just wrap it with an effect Handler right that when it intercepts the the random effect it Returns the value zero right okay yeah and now we continue the the test the code so now random that per performing of that effect gets sort of replaced by the value zero and you continue the the execution of the program uh you can see how this can be used to mock things in some ways
right yeah it can also be used for example to replace um the kind of random numers you have right like you could go from something that's pretty naive to something that is like cryptographically safe right without changing your program at all so it's something that needs an external key to be plugged into the system to yeah yeah yeah like it's it's a I think it's a super interesting way of detaching how the effects are performed right from where the effects are needed yeah and that lets you lik
e customize your program in interesting ways yeah there's a hint of dependency injection in this yeah yeah plays out differently in code yeah like it doesn't look as much as as well I I will declare my dependency it's more like you you will have a function called random that under the hood just calls perform this effect right yeah so to you the user of this Library the random Library you're just calling random right but the provider of the library gives you this effect Handler that you wrap your
program around and this is why you'll see a lot of the newer concurrency libraries for Camel coming with a top level like run function because that function installs the Handler it's like saying you know we we add that TR catch around your entire program right right so it's like building up the program and then when you're ready to run it saying in what context it runs yeah exactly yeah okay okay I'm with you I'm with you so those two things why do they so threads and effect handlers why does t
hat lead you to think now is the time to start pting my favorite bits of erlang to O camel right um yeah so the main the main reason is that we didn't need the multi-threaded part right to do this but now that we have it is we can do it really well right we can actually uh to for some use cases maybe even be competitive with the earling BM which I I think it's really interesting um effect handlers allow us to say you have a function which will be your process right and when you want that functio
n right to receive a message for example right we will intercept that effect and we will suspend your process which means we will not IM immediately continue right the execution of your process your function okay and that allows us to say as a function executes and a function corresponds to a process so it could be like a record function for example right as it executes eventually we will know when it's time to say all right enough suspended let's let's let a different function run now oh so eff
ect handlers allow you a way to write something like an await keyword as an effect and that cracks open the let's make a nice actor model inside o camel I'm with you yeah yeah yeah okay that's okay so take me through what you're doing with that how much of erlang are you trying to build that's uh yeah um so I can tell you what we're not trying to do that would be a shorter list okay so like we started we started uh the riot maybe last September right and it was like okay let's let's see if we ca
n make a a process right yeah the the core abstraction of Airline if it suspends we can resume it great let's see if we can send in messages right and we implemented the message signal so now two processes we have a function called spawn you call that with another function which is the body of your process right and then you get back when you call that function you get an identifier from the process so you can use that to send messages to it right yeah and and um we managed to make that pretty f
ast as well like you know spinning up process is just allocating at like 140 words a tiny data structure right um it's even smaller than in erlank if I can brag about oh really because the actors are famously low overhead in llang yes well done thank you I mean it's still very early stages and it doesn't do like a third or like a tenth of what LL does but you know uh it would be interesting to see if a year from now two years from now it's still that small and that fast but this is this is where
we are now right yeah being real um so we got messages between processes and we want to type messages so we made some design decisions there right messages are also just data and because data by default is immutable in no camel then sharing that data is pretty cheap there are edge cases like what happens if I make a message where I have a mutable value inside of the message you can do that don't do it but you can do it right like otherwise you'll send a message and then by the time it arrives y
ou would have changed and that sounds like a terrible idea right yes um yes exactly but this stuff gets much easier when everything's immutable exactly like if you cannot change stuff like in ourl it becomes much much easier yes um so we made that and we made that fast and um then we said okay now we actually want to know when two processes are linked together like if one process needs to know that other process dies how can we do that and we implemented monitors which is a feature of the airlin
e VM as well that allows one process to receive a message whenever a process in monitors dies like it's terminated right and and then we also implemented links which also it's like an extension of that that says if you link two processes together and either of them goes down the other one goes down as well right ah yeah yeah so now you can start to build up this supervisor tree thing corang has yeah so now we said okay we have we have monitors we have links let's build supervisors right yeah wha
t's a supervisor it's just a process that creates more process proc es and monitors them right and then has a strategy for deciding how to keep the other other processes its child processes alive depending on how they die right you have many strategies I think the only one we implemented right now is one for one so if you have a supervisor that has like four children and one of them uh sort of stops then it will just restart one children right but we can Implement other strategies such as one fo
r all where it's like one of them goes down everything goes down the musketeer pattern exactly yes exactly literally we might call it that that's a good name uh so we we went there and then once we had supervisors right we said okay so what else can we build here can we build gen servers right the fame earling obstruction for Building Services inside of the earling VM and it's a little harder slow down a bit because gen servers I'm not 100% sure I know what that is so a gen server stands for gen
eric server and it's a pattern inside of erlank remember we talked about the OTP the open Telecom platform right yeah it has a bunch of things like a bunch of things that you can just reuse to say oh well I'm going to need in my application which is going to be like a microservice like tiny universe right yeah I'm going to need one server for users and that handles like user requests and one server for like I don't know orders or things like that these are not like HTTP servers they're just like
airlink servers right yeah and the pattern of saying the server needs a que of requests that come in and it may receive requests asynchronously or synchronously such as send wait respond or just send and forget that pattern um with an internal state that is carried around in server it is just very very common in the systems So eventually the airl OTP Library included this thing called JN server which allows you with very little um overhead right to build one of these servers and it manages stat
e for you and it manages all of the timeouts for the different calls it has a bunch of logic implemented for you right yeah yeah so you just Implement like oh what do I want to do when this specific request comes and then you put your logic there and that's it right right yeah it's super powerful Elixir also uses this a lot right um but it works it's I think it's powerful and easy to use on their end because it's not typed and okay yeah when we started typing it was like holy crap this is this i
s hard like there's a lot of tiny little things we need to do here some maybe even unsafe code inside of okam right to make sure that we can provide typ saave apis to a user while still being flexible take me into that cuz it's from what you described it doesn't immediately sound like typing is going to make that hard so there's a couple of design decisions right that we've been making one of them is that there is a single uh message type right the entire system that work per server or for the e
ntire system for the entire system so there's literally like a riot message T that's type of all messages in Riot right okay right and it's an extensible type uh this is one of the trade-offs that some people uh that some people really don't like and some people go like oh yeah that sounds like we can run with this right okay in my head it's like the right amount of type safety we need right now in the future we may change that right now we're okay with this okay basically if you need to create
a new message a message type you can add a new Constructor to that type so then the entire system knows how to handle your messages now and anybody that has access to that Constructor right is able to create new messages of that kind and receive and like patter match against messages of that kind does that mean I mean in your example earlier you were saying you've got one server for users and one server for orders does that are you saying that the orders server will now have to know how to handl
e user messages no it doesn't have to right um it may be by the definition of like the receive function that we use to receive messages in a process you may receive any message of the type message right yeah so you just don't know what message you get unless you patter matching the constructors um right so when you build your order server and you call receive yeah and matching that like you get the pattern match there you could P match on users messages right I don't think you'll get them I don'
t think you'll be sending those messages to the order server right yeah um this happens also in a and elixir where you can send whatever message to whatever process right yeah yeah um and we have two ways to deal with this right now the first one is that the constructors themselves have visibility rules in OK cam okay so when you build your module for users you can say oh these are the you know create user or like delete user messages right yeah and they're private to this module so ah have a wa
y of guaranteeing a development time that user messages will be scoped to the user services so you may still receive any message right but you can say I only care about this once okay and done um one I think one thing that we're doing with Riot that goes a little bit against the ethos of ok caml is that with Riot you can let your program crash and that's okay yeah because you've got this other mechanism for that right exactly yeah okay um but the second thing we're building and we had it in an e
arlier version but we removed it and we're going to introduce it again is the ability to write message selectors so while the message type is large and has many Constructors right you could specify a selecting function that says whatever message we have in mailbox right only give me the ones that fit this pattern or this function right yeah like a filter like a predicate yeah yeah um and that would allow us and this is a pattern that um I thought it was a bad idea and then I saw gleamy doing it
and I was like okay maybe it's a good idea so Lou thank you for that uh so we're gonna bring it back and and it allows you to say from this massive type of messages right you can constrain down to a very very small like sort of space maybe even a single message right that you really really care about so these are sort of the the ways which we handle messages that makes sense so you can narrow it down to just the domain you just do it slightly different way than having several message types exact
ly exactly yeah okay yes I've always wondered with this um I didn't know it was called gen server but I've always wondered inside of erlang how so in this case inside of oaml how you are managing the under the hood the mailbox thing and the statefulness of the server and persistence thereof yeah AB about that yeah I wish I could just like share some of the slides I had for the presentation no slides no no slides no slides so can I draw in the air and then users pretend that they see what some pe
ople are listening to this on Spotify so you just have to paint pictures like radio paint the pictures so imagine a square from from coordinate zero to 100 realistically right like um a processing in Riot right is a data structure it's essentially a record or you know struct that has a couple of of interesting pieces of data it notes his identity it notes its identifier right he knows the scheduler it currently runs on and it also contains two different mailboxes right okay for all intents and p
urposes they're the same they're just used for like when you're selecting messages sometimes we put some of the messages that didn't get selected into like a separate mailbox so we can read them later okay but the mailbox is a lock free queue and when you are going to send a message to a process you only have access to the identifier the PID uh the identifier is essentially an opaque integer you don't really know that it's an integer but it's Unique for the longevity of the entire system if you
restart the server again like the whole application then you know another process will have will reuse an identifier a so they're not unique uh like you know across program exec utions they're just unique during a program execution right yeah yeah that makes sense and that PID is used as a lookup so there's a small lookup into a big hash table right yeah where we say find me the process that belongs to this bit and then we get the structure which is the process and we say all right let's access
the mailbox of that process and we'll put the pointer to the message I say put the pointer you don't do that but OK camel does that right um in to that Queue at the end of the queue yeah there has been um thoughts like I've been thinking about how to make this even more straight forward so maybe the PID actually is a pointer to the mailbox directly so you just like put things in there yeah okay I can see that which could be faster but right now like we're not that it's not a concern right now li
ke message passing is not really the bottom L right now okay uh so that's a little bit how it works so you make any data structure any piece of data that fits the m message T type and then you call the send function and you put the process ID and the value the message yeah and then we put that at the bottom at the end of the queue of the mailbox okay that seems straightforward what about the state is that just is that just a chunk of memory uh well we don't really handle that in Riot right like
Riot doesn't have access to the the current state of a process the way that we do that is that when you spin up a new process and we allocate the set data rure we allocate a continuation right we allocate essentially um stopped function like a whole stack Trace that ok camel gives us the ability to stop and resume right and that's completely opaque to us so if you want a manag state within your process you need to build a recursion for example yeah so it's not uncommon to see that you have a spa
wn function right calling to a function init and that init function set up some initial state and then it calls something like loop with that state right yeah and then that state in that Loop just continuously you know Loops forever essentially and in in its body the loop function may do some things like for example it starts by saying I need to read messages first so I'm going to call receive right and then I'm just going to wait there until a message arrives and then when we get a message like
add user then I will access the state which is just a parameter to the loop function right yeah and do something with that maybe update that state and then call loop again right to yeah go back correct and this is also the erlang way of and The Elixir way of handling and the Glen way of handling State then what do you do about persistence between runs is there any notion of I erlang Has like this built-in database system doesn't it yes save your state too if you need correct erling has three da
tabases you have a ETS Airline term storage which is in memory you have D DS which is disk based Airline term storage okay and then you have amnesia and Amnesia Builds on top of um DS and he adds a lot of things on top like the ability to do queries uh do transactions and stuff like that okay uh we have started building something that we're just calling store right and that is kind of like an aamel term storage right right we do have a problem problem which is that we don't have one type for all
OK camel values right so we have to recur to a common pattern in OK camel which is functors uh which is different than the typical functors people talk about from category theory in hasal the typical functors that people talk about from category Theory like they're in coffee shops they're talking about category no it is not free range functors these are like you know lab growing F fun yeah what is what is an oaml funter so an O camel funter is a a very interesting thing right it's a function fr
om a module to a module so it takes a module as an input and produces a new module as an output okay and uh it's it's kind of crazy it's a great way of doing um sort of abstractions and patterns because you can say I need a store right yeah and I need the store to be a module that handles some kind of key type so you can like put strings there for example and some kind of value type like for example integers so now we have this like table this store from strings to in right yeah and you need it
to always have for example a get function and a put function and some other functions in there right yeah makes sense but you would like the store to be able to be customizable a little bit for example change the type of the value or change the type of the key right y um you could try to build this such that the module is generic right and then that means you get um the two type parameters inside of a generics to a larger type which is the type of the whole store right yeah uh but that means you
maybe cannot specialize certain things like for example how do you do the Lup right of um of a key right because now you don't really know what the type is it's going to be parameterized when someone creates a new store and if you use a functor you can say I will give you a store if you give me a key type right and a hashing function that I will H use to find values of that key type match them up together yeah and then you create a store that is very optimized specifically for that specific uh
key type you give me and um this is kind of weird to like explain I think without code but it allows you to say I'm going to give you a specialized version of a module right giving some inputs and the input happens to be another module with some functions or type definitions yeah okay I I can see why you why you want to take a more generic package module set of functions and make a more specific one I'd not considered modeling that as a function from module to module but sure neither did I yeah
okay yeah this is just something I learned and I was very mad at them at the beginning because I didn't understand and I I think I understand better now but I also try to avoid them because functors are powerful but they're one of the type system and you know language features that are hardest to sort of um let's say get up and running right and really grasp they have a lot of gochas they have a lot of tiny things that are complex like if you have a type inside of the input module and you want t
o have the same type in the output module right yeah and you want the type system to know that these are the same right you also have to like have this annotation that tells the time system this type of quality escapes this sort of module closure and it's just there not the best developer experience to be honest okay yeah yeah anyways so we're using one of those to be able to create these stores so that you can say I want to save users okay we give you a very easy way to create a user store righ
t yeah and then that means you don't need readi you can just use that thing right okay yeah okay that makes sense while we're talking about persistence then I have often wondered and this is my background in Kafka coming through I think that um you have mailboxes in this system but if the system and the system is expected to crash if it crashes it loses the whole mailbox yes and I keep wondering can we not build a version of erlang or oot or the beam where the mailbox is more persistent and if I
crash I can just replay the messages I lost since the last time I S saved my state yes I think we could um the volume of messages though can be pretty big because the messages may be very tiny and sometimes it may be very large right not necessarily the size of the message but let's say the the meaning of the message for for example if if I'm going to send in erlank a string the string may be like a really large piece of data right yeah like potentially even gigabytes but because earling is sup
er optimized for this use case then your just really passing a pointer around which is nice right yeah um if you were to save that then you will need um let's say a persistant like I don't know like a transaction log right for the message but you will also need to be able to persist all the data right that's associated with a message including potentially like you know four gigabytes of XML that yeah okay and that is like we wouldn't do that in Kafka either right we would not send a 4 gigabyte m
essage right you would put that somewhere else first and then you will send the message and then Kafka will persist the message and when you get it then you can go to the persistent store for the rest of the data and get it right yeah yeah yeah um so I think that it would be hard to build like like a one siiz fits all solution here um it would be possible to do if you have like a very strict check on message size like if you say we will build a version of Riot that is meant to be persistent but
also running with like tiny messages right right yeah yeah so if you try to send a message that's big we go like nope you cannot send that so there is a future where you could have like an optin persistent message CU provided you knew that it came with caveats yeah exactly like I I've been thinking about why would it take to make Riot a little more you know we could build soft realtime systems in the same way that we could with arang but we have a lot of things that arang doesn't have which I th
ink makes us a little more flexible here like for example CLI Tools in IRL and elixir have a problem and they're slow to start right because you need to boot the entire machine and you have all of the setup for the schedulers and everything yeah yeah and this is a system that's expecting to run for 10 years at a time right so it's completely fine right like if it's going to run 10 years 100 500 milliseconds of boot time nobody cares right yeah but um I've gotten used to having fast startup in th
e okam ecosystem and its tools so Riot ends up having a really fast start of time right within 7 milliseconds or so in a you know modern MacBook you have the tool running and you can have something running across all your course already so that enables some things right that Airline just cannot do right now and by Design maybe never maybe doesn't even care to do which is fine uh but it also means that we have less overhead in some things like if you were to do more let's say CPU bound work I thi
nk OK camel is better suited for that than irang is right um primarily because he compiles to native code directly right and like ahead of time so we can do a lot more optimizations there okay yeah the airling Jeet is like fantastic right and the work Lucas LaRon has done with it is just just incredible right yeah but it's still not designed for CPU bound um workloads right it's more IO bound and while Riot is iob bound as well I think it'll be interesting to see how how far can we push it down
to the real time like actual real time um workload space that maybe are more CPU bound because the overhead of running processes is smaller and the startup time is also smaller and things like that are we talking like hard real time there like you've got 50 milliseconds to answer this question or die I think to do that we will need to do some work you know camel itself oh okay because right now we don't have a way of knowing like the only point in which we can jump between processes for example
yeah is when we have an effect right yeah so you'd have to guarantee without any change under the hood you'd have to guarantee that any actor did actually call A A wait in a reason amount of time exactly yeah yeah we have an effect for that that we call yield Y and if if you write an actor in erlang and you do an infinite Loop like f calls F infinitely yeah erl doesn't care it's like it will just run and you can run anything else and it will just schedul it properly it was just a little bit of y
ou know dead CPU time there that's it for us if you speed up an actor and you do wild true right yeah then you have breaked that scheduler [Applause] right yeah so we have this effect that it's just a function called yield that you can call anywhere right and ideally you can call in your hot Loop right so that you tell the scheduler hey I'm doing a little bit of work at a time like please consider descheduling me consider consider contact switching to somebody else ah so you imbue your function
with politeness correct yeah some people call it Cooperative I think polite scheduling is a better ter for that yeah so they're they're not so eager or greedy right yeah but before I before I get back into my while loop for the billionth time do you want to do anything yes yeah exactly would you maybe run for a millisecond before I take over again yeah um and I think that works okay it will it would be much better if we could have preemptive scheduler you know like scheduling so that even if you
have a while through then we eventually deschedule you right like you run for more than 15 milliseconds or 15 micros seconds or whatever then go go away but that definitely seems like it would need language changes I think the main language change we would need is not that big um it's almost like a compiler plug-in right like if we could inject a CO to yield for you right at let's say the boundaries between function calls right with gravy right that will be all that we need to do okay yeah yeah
we haven't explored much what that would take because we haven't needed it yet but I think as eventually one day in the future Riot becomes more stable and gets closer to like a version one right yeah then we would definitely want to have something like that so people don't have to worry about like oh no I call this like you know recursive function through these all the three functions that I didn't realize right you don't have to think about that it just runs well yeah infinite wild Loops are
easy to spot but when it happens that a BS CS a that can be harder exactly yeah okay well it sounds like you've got plenty of time to campaign for it to be in oaml 6 probably yeah that's a very good idea actually I had thought of that so thank you okay cool um you've so we've talked about multiple CPUs I am wondering because I think this is a thing llang does and I don't know if you do it in on um in your project I don't know if you're planning to can't you direct messages to an actor that might
be running on a different server o good good question so Riot one of the nonn goals of Riot is distribution oh okay and in irine distribution means you have different noes running and you connect them together and they get connected essentially by a third party program to the BM right that's called EPMD airling Port demon Port manager demon which sort of keeps it track of where other airl nodes are running right and it tells the DM essentially do you have any messages for this port right for th
is other node yeah okay I'll send a message through that is not something that we couldn't do we could do it but to guarantee type safety across the nodes that's a problem right yeah I think this is where I would say we don't want to do that and we want to stick a queue in the middle like yeah yeah just use CFA right that sort of thing that makes sense so you've just raised another problem then because type safety across nodes has almost the same problem as type safety between runs on the same n
ode yeah like if I bring my whole system down and bring it back up what What mechanisms have you got to deal with that because because I need to deploy new code but the Assumption of the whole llang way of writing things is that the the master process will run for a very very long time U I think this is more like a general problem right like it would happen in Java it would happening like JavaScript yeah like if you have an external queue or a database right and you changed the data model there
but you didn't change your program to deal with that or vice versa then you have the problem at list in know camel and type languages we have the ability to say the user password change from like I don't know um an integer to a that's a terrible way of saying it the ID change from an integer to a uid right yeah like a string um also probably bad like just start this uid as a string but anyways I think it's like as an example as a toy example would be good right if you were to change that um you
would have a way of knowing how your application reacts to that change right before running it um so I think we have an advantage in that sense from untyped languages where you would have to sort of try and and have tests for these things right whereas you know camel you just change the type recompile and it shows you this is everything that broke right so you already know what to do but guess what I'm getting at though is do you have any plans for hot deployment o okay yes um no that's also a n
goal okay yeah so n definitely n goal it would be really hard to do uh I think it would be fun to do and we might be able to do some of it like there there were discussions at some point in the airling ecosystem foundation work groups right around type safety of how could we guarantee some of this and I remember I used to work at airling Solutions and I had chats with Rober Bing about this as well and uh how would you do some of this right you probably want to have like let's say a hash of the
information of the type right so you can compare whether this thing has changed or not and you will want to be able to have a transition function from the old type to the new type right yeah to sort of guarantee that that's there um but it just sounds like more trouble than it's worth in the same sense that I don't know a lot of people that do hot code reloads in irank or Elixir oh really is that an oversold and underused feature are you saying no those are not my words but sorry yes no no worri
es uh I I think I think that it is a great feature and if you need it then it's just fabulous that it's there right Clara I remember like actually copying a bite code like a Beam file into a machine right and reloading that manually and just to be able to debug something and it was just fabulous uh but for most I guess for most systems what we have most of the state outside of the program we try to build them stateless in some way right yeah so it's easy to say well if it's saved on a database t
hen you just like shap a new container right and it will be picked up uh because it does have the same problem it doesn't have the type level problem but you still have the problem that the type level problem surfaces early when you have the types it's like you change the type now you need this transitioning function between Type A and type one and type two right like two versions of the same thing erlang and this is also encoded in OTP because clearly they have had this problem because they had
the use case where you you couldn't bring back the the box and reprogram it you had to sort of update the code there right while it was running you have this idea of a code uh code change so you get a function right that has the information about the new code right yeah or rather the old code and the old state right and this is a function the new code must Implement so you get a chance to do the Handover between right yeah the two versions of the same module right on a single process and this i
s like you know we're talking you're in space and you need to sort of update a module and here's the new copy of that and the program kind of knows how to morph the data from one it's just fantastic right it is like scii almost right yeah it's making me think of a scene in um The Martian yeah yeah yeah but yeah it's I guess it's the same problem kafka's wrestling with this you get it any way you're doing schema migration right right exactly you're going to have to deal with that yeah yeah okay s
o I've uh I've challenged you on two non- goals tell me where tell me where Riot is at right now and where you're taking it next so uh one of the things that we've been diligently almost like obsessively doing with Riot is build vertically so we started building with the idea that we want to have an entire vertical stack from the bottom from like conditional compilation in a camel that is done nicely all the way up to we have a a web server that does something like live view for Elixir so we hav
e essentially socket socket driven or socket based uh Dum patches sent to a website okay yeah yeah and to do that we build literally everything in the middle right we build the new versions of of strings for ok caml that have pattern matching like you would have in Elixir Aang we build socket pools HTP server websocket server so new sort of protocol implementations for those things and um right now we have this phase one of the project which is meant to have a stack that can do that right right
it might not be the most efficient it might not be the most complete right uh it would support htp1 and web sockets right it would cover some use cases so if you want to build like your next U sort of toy app side project with it you can do it yeah that would be like phase one uh on the next phasis hopefully we'll be able to to cover more ground support maybe http2 give you an actual web server that has all of the bells and whistles that you expect out of something like plug on Phoenix on Elixir
right yeah and at a lower level of the the runtime we're hoping to get some things like uh process steeling working correctly and what's that so Pro I I love that feature uh so um process still still ceing was originally built for Riot a couple of months ago but we have a couple of bugs the idea is that once you being a process in a scheduler other schedulers that don't have anything to do can actually steal that right so you distribute the load across all your cours automatically okay and that
's um and it's that way around it's not like the server saying I'm over loaded you take this it's it's the quiet threat quiet CPU saying well I don't have enough to do I'm taking that task from you exactly like if you're busy doing something right now you probably don't even know how many processes you have to run right and you don't want to like say hey does anybody want a process please take it from me it's it's the other processes that are just idling that can say well you know you seem to ha
ve a lot on your plate so I'll help with that yeah um yeah and it has a couple of subtle bugs in it so we had to roll that feature back but the nice thing about it is that it it helps you say doesn't matter how many cores you have doesn't matter how many um processes you have right if you just keep spawning them to do work tically of course right the yeah the process here key eventually you will distribute them across all of your cores and the load will just be as fast as we could possibly make
it yeah and all low in the middle of course but yeah yeah but but that's nice that you could it would be self L balancing in that way yeah yeah yeah and uh I think I think you can start to see with this couple of features like process that are suspendable message passing link supervisors right process dealing across all the different schedulers yeah how we are bringing we're really bringing the airling VM to a camel yeah and keeping the type system while you're there exactly that'd be nice yeah
yeah yeah I tried to do the opposite before I tried riding caramel and bring the OK camel type system to erlang and that was hard so I'm actually really happy that Louie just charged forward and gleen V1 was released recently I was so excited over it that's the way it should be right it's all shipping ideas from one language to another until we're all in the right place until we all write you know this mix of ok onlink yeah yeah with some hin Idis on the side please uh maybe sometimes if really
necessary Okay cool so where are you at for the future is this something where we should be encouraging people who are interested to try it out to come and join the team and submit absolutely so we have three ways in which you watching on YouTube can contribute uh the first one is uh use it right build stuff with it grab Riot um the the okam ecosystem has a little bit of a rough start for New Commerce so be patient with that we have Discord forums where we can help you with anything that you're
experiencing there but build things because when you build things bugs come out and we get to fix them and we get to make Riot better yeah now if you already ride a little camel or have um are enthusiastic about learning more camel and writing things then you can also join some of the projects right we have about five different GitHub Orcs for the different parts of the stack with some repos each of them and we have about 25 to 30 collaborators at the moment on all of them which I am super excit
ed about by the way decent number yeah I I'm most excited about the community we're building around Riot than Riot itself it's like yeah yeah so there's a lot of uh good first issues there that we are trying to put we put a lot of car into that to make it easier for you to pick up and say okay this is the problem this is uh some implementation notes you know this is maybe how you could test it right so we can give you that package and you can just run away and implement it and come back to us an
d process and uh contribute in that way and then of of course the last way of contributing to Riot right now is to sponsor us either uh on the streams right when we're like essentially doing uh subscriptions I believe that's yeah Subs uh because we do a lot of live stream development for Camel and Riot right and also get up sponsors which we recently crossed like the 31st 32nd sponsor and power of two love it yeah ex really good number right we're hoping for the 64 soon uh and actually like uh l
ast week we had one one sponsor drop a thousand bucks a month and I was like holy crap someone is really batting on this right yeah we even have one user of Riot that it replaced a small airling service with Riot and we're waiting for approval from the company to be able to share more information about that right but I am I mean I'm terrified because it's so early right it's like no please what are you doing but also excited because it shows that maybe there is a subset of Riot right that is sta
ble enough for some workloads right now which I think is interesting because I don't know what subset is just yet right yeah that's a really interesting time for a project I think so I think so I'm really excited about it yeah well I probably should leave you to get back on with it and get coding and feed sounds about right I do need to feed Leia yes I don't know she she's there somewh so Leandro thank you very much for talking to us and I wish you the best of luck on the road to Riot version on
e thank you so much chis for having me uh I will send you uh emails with people that you should definitely bring on board to talk about like milange camel and things like that right but again thank you so much for having me and I hope you have a great weekend you too Cheers Cheers do you know I'm kind of jealous that is a great time to be at the heart of a project it's almost better than when it's successful it's that period before when it's really starting to take off thank you Leandro and I wi
sh you the very best of luck for what comes next as we said if you want to find out more if you want to try Riot if you want to support riot or if you want to take a look at the other languages we mentioned like Reon ML and rescript all the links are in the show notes before you go and check there if you've enjoyed this episode please take a moment to like it share it and rate it I always appreciate the feedback of course and it helps other people discover this episode too and if you're not alre
ady subscribed consider subscribing now because we'll be back soon with another great mind from the software world and possibly another the guest cat I can't guarantee it but we'll find out soon and until we do I've been your host Chris Jenkins this has been developer voices with Leandro estera thanks for listening
Comments