Welcome my video about my presentation at the Kotlin London Community. Though online, the presentation went amazingly good. This session was about practical uses of Project Arrow that I found in my company to work for real life scenarios. Because it is not possible to talk directly about the actual code used in the project, I created a parallel world where we can use Project Arrow. Project Arrow is a functional library that we can use with Kotlin, to make sure that we use best practice related to functional programming theory. The idea is to maximize the potential of a functional code and adapt it to use with Kotlin coroutines as efficiently as possible. The initial idea was to give the presentation live and present in London, however, due to logistical issues, it was not possible to get everything managed on time. Still, it was an amazing crowd, an amazing audience and I just want to say thank you to David Craft for making this possible for me. Find all the necessary information about this meetup below in the description. For now, be sure to stay tech, keep programming, be kind and have a good one!
---
Chapters:
00:00:00 Start
00:02:11 Introduction
00:14:46 Overview
00:16:36 Functional Data Modeling with Arrow Core
00:20:42 NonEmptyList (NEL)
00:30:32 Error Handling with Either
00:34:14 Error Handling with Either in a cumulative way
00:39:12 Using option and nullable to handle non-existing results
00:50:45 Arrow Optics - The theory and practice of Lenses
01:03:21 Resillience with Orchestrated Saga
- https://dev.to/jofisaes/newscast-using-sagas-with-choreography-and-orchestration-patterns-kotlin-example-3ff6
01:12:09 Asynchronous data with Arrow FX - parZip
01:20:41 Suggesting STM video:
- https://youtu.be/DmV0bakYNps
01:20:54 Consistent state management with Arrow STM
01:29:33 Conclusion
01:32:56 Suggesting STM video:
- https://youtu.be/DmV0bakYNps
01:34:05 Questions - The Inquisitive Cat
01:36:27 Resources
01:36:51 About Me
01:37:10 Thank you!
01:39:51 End Note
- https://youtu.be/S3k6C1XaYr8
01:40:37 See you in the next video!
01:40:48 End credits
01:41:27 Disclaimer
---
Source code:
- https://github.com/jesperancinha/space-ship-adventures
---
Meetup Info:
- https://www.meetup.com/kotlin-london/events/306456259/
---
Sllides:
- Scribd: https://www.scribd.com/presentation/846009877/Boosting-performance-and-functional-style-with-Project-Arrow-from-a-practical-perspective-with-Spaceships
- Slideshare: https://www.slideshare.net/slideshow/boosting-performance-and-functional-style-with-project-arrow-from-a-practical-perspective-with-spaceships/277622766
- Speakerdeck: https://speakerdeck.com/jesperancinha/boosting-performance-and-functional-style-with-project-arrow-from-a-practical-perspective-dot-dot-dot-with-spaceships
---
---
Chapters:
00:00:00 Start
00:02:11 Introduction
00:14:46 Overview
00:16:36 Functional Data Modeling with Arrow Core
00:20:42 NonEmptyList (NEL)
00:30:32 Error Handling with Either
00:34:14 Error Handling with Either in a cumulative way
00:39:12 Using option and nullable to handle non-existing results
00:50:45 Arrow Optics - The theory and practice of Lenses
01:03:21 Resillience with Orchestrated Saga
- https://dev.to/jofisaes/newscast-using-sagas-with-choreography-and-orchestration-patterns-kotlin-example-3ff6
01:12:09 Asynchronous data with Arrow FX - parZip
01:20:41 Suggesting STM video:
- https://youtu.be/DmV0bakYNps
01:20:54 Consistent state management with Arrow STM
01:29:33 Conclusion
01:32:56 Suggesting STM video:
- https://youtu.be/DmV0bakYNps
01:34:05 Questions - The Inquisitive Cat
01:36:27 Resources
01:36:51 About Me
01:37:10 Thank you!
01:39:51 End Note
- https://youtu.be/S3k6C1XaYr8
01:40:37 See you in the next video!
01:40:48 End credits
01:41:27 Disclaimer
---
Source code:
- https://github.com/jesperancinha/space-ship-adventures
---
Meetup Info:
- https://www.meetup.com/kotlin-london/events/306456259/
---
Sllides:
- Scribd: https://www.scribd.com/presentation/846009877/Boosting-performance-and-functional-style-with-Project-Arrow-from-a-practical-perspective-with-Spaceships
- Slideshare: https://www.slideshare.net/slideshow/boosting-performance-and-functional-style-with-project-arrow-from-a-practical-perspective-with-spaceships/277622766
- Speakerdeck: https://speakerdeck.com/jesperancinha/boosting-performance-and-functional-style-with-project-arrow-from-a-practical-perspective-dot-dot-dot-with-spaceships
---
Category
🤖
TechTranscript
00:00Make up lenses from Aero Optics.
00:02With only a few simple lines of code, you can modify your data without breaking a sweat
00:07or your brain.
00:09Say goodbye to boilerplate code and say hello to composable clean transformations.
00:14Lenses will make this feel and look like a walk in a park without having rabbits popping
00:18in from everywhere.
00:20But before we continue with the magic of Aero and Optics, we first need to know how to install
00:26this.
00:27And the way to install this is basically by using the KSP plugin, which needs to be
00:31installed in our plugin stack.
00:34And then we also want to have the KSP activation installed in our dependencies.
00:41Then we just need to use this in our dependencies in our build Gradle KTS file, or if you are
00:45just using Groovy, just build Gradle file.
00:49We also need to make sure that we are using the implementation that can interpret the Aero
00:52Kotlin code, because there will be code generation.
00:56And that is the IO-KT Aero Optics.
00:59So essentially, we've got a combination of three different important things that we need
01:05to configure in our build Gradle KTS file.
01:07One is the KSP plugin that is the Kotlin symbolic processor.
01:13And then we need to have our optic dependencies in our dependencies.
01:17And we also need to activate the KSP plugin so that code can be generated.
01:22So what is this whole thing about code generation?
01:24Essentially, we are creating lenses for our data entities or DTOs, wherever we apply that
01:32annotation with optics.
01:34That means that Aero Optics in combination with this plugin, particularly and not exclusively
01:40this plugin, there can be other plugins that can do this for us.
01:43But when optics gets interpreted, there will be code generated, that is the code that will
01:50allow us to use lenses so that we can access the different levels of our deeply nested data
01:55structures.
01:59This is our data model.
02:01And our data model is the one that we have seen before, it's the same thing.
02:05We've got transmissions ngdto with a message package, and the message package contains a
02:10list of different messages with a timestamp included in the package itself.
02:19As we can see here, Optics this annotation is important to apply to every one of these classes.
02:24For each one of them, we will generate lenses that can allow us to access to them from the
02:30parent object.
02:34So here we go, we've got two different lenses.
02:37When we run this code, and we build it, the generated code will give us these two lenses
02:43transmission ngdto message package, and transmission ngdto message package.
02:50What can we do with this?
02:52Let's see further on.
02:54We have these two lenses, and let's use for example message lens.
02:59And by using message lens, we are now able to access our object, transmission, and we
03:07are able to get just the message package.
03:10So that means with a simple get, we can get the transmission, the full transmission, and
03:15just with that, we get just the message package.
03:19Maybe here we don't see exactly what is the advantage of using this.
03:24But whatever lens we have had in any of the deeply nested classes that we have from the
03:31parent class would be able to be accessed with a simple get.
03:35And not with dot, dot, dot, dot, dot, how many dots we would want.
03:40But that would be much more complicated than what we see here.
03:46And with a simple get, we get that data.
03:48But this probably is not the most interesting thing.
03:51Because of course using dot, dot, dot, dot, dot is something that we sometimes do instinctively
03:54in our code.
03:56The best part is the set.
03:58And so here we've got the same lenses, and we are going to use the same lens, message
04:02lens, in this example.
04:04So if I use message lens set, and say that I want to affect the transmission object, and
04:11set the list of messages to be simply a list with one single message that has purpose, phantom
04:20purpose, and message phantom message.
04:23What this will do, it will create a new transmission ngDTO that we can send back to the client that
04:30has the purpose in the message replaced by another message.
04:35And there will be a non empty list of only one element, which is this element that we
04:40see right over here with ID zero and package ID zero.
04:45So in one single line, we were able to change that object with the specific property that
04:50we want to change, because we've used a specific lens for that specific object.
04:56So now it's time for us to have a look at the example of this code.
05:02Right over here, we first can have a look at how the build is being set up.
05:06So if we go here to the build, we will see here all the way up that we've got the KSP plugin,
05:14the Conflin symbolic processor.
05:17And then here all the way down, we've got the activation of the KSP plugin.
05:23And here we've got arrow optics also being used in our build file.
05:28This is important because only this way we can get optics to be generated.
05:32Now, this project has already been generated multiple times.
05:35So we should be able to find here in the build, the generated objects created by KSP.
05:41And these are the objects, message objects, message package optics, and transmission and
05:47GDTO updates.
05:49And if we go inside all of them, we will find getters and setters, we will see a lot of different
05:55code that is a Kotlin code.
05:58And we can see that this will take care of all of the different ways of accessing the
06:03deeply nested structure of our code.
06:06It will also include other things like lens, optional, traversal, and there will be lots
06:13of different things here that we can use.
06:15However, the most interesting part of it is to be able to change an immutable object, well,
06:21not change the immutable object, but to create another immutable object that is a copy of the
06:26original one with a specific property changed.
06:29And when we use set, the result is always the same object that we are trying to affect.
06:36It's just that it is a set from a lens to a property that gives back a copy of that whole
06:42entity with that property affected from the value that we put in as an input argument.
06:49All right, so let's see now how does the code is behaving.
06:55And if we go here to app source, and we go here to, as we have seen before, the route,
07:03the spaceship routing, we see different objects over here.
07:08And one of them that we are interested here is this part over here.
07:12Here we've got the post endpoint, which is called the phantom endpoint, where whatever message
07:18we send, we will be using the lens that we have created here above.
07:24And this lens is created from those generated files with the arrow optics structure.
07:32And here it uses KSP to create there.
07:35And then here we've got our lens that we now use in our endpoint to affect the message that
07:43we want to give back and give back a phantom message.
07:46Let's see this in detail.
07:47And let's now debug our code and see what is happening here behind the scenes.
07:53So the service is already running.
07:55So if I go now to the example file, and here I look for phantom, because we've got all different
08:10examples in this file, of course, and we want to make sure that we run the right example.
08:15So this one is located in the route pieces and then phantom.
08:18Let's double check that if that matches exactly.
08:21We've got here one route, inside routing, and then we've got pieces and then slash phantom.
08:27So here we are going to land in this piece of code after reading the transmission.
08:31Let's do that.
08:32Let's run it.
08:33And now we've got transmission.
08:34And then here in transmission, if we run the debugging, we can see here transmission,
08:41we've got the result that we are sending, we've got the message package here with the different
08:47messages, and then there is only one message, and the message has purpose, message, and the
08:54message bcc and cc are nulls, and the rest is all null.
08:57It doesn't really matter, because we are only just giving back a copy of this object with
09:01just a phantom.
09:02Now imagine, if you wanted to change the messages, we probably would have to make a copy of the
09:07whole entity, so we would have to do something like this, let's see, transmission, copy.
09:18And then here we would have to do something like message package should be the transmission,
09:29and then message package.
09:31And then what we also want to do is copy and then just say that messages equals whatever
09:36message we want to give it.
09:38And as you can see, this is like one dot, one dot is a copy, and there's no guarantee that
09:45we didn't miss any of the steps needed to create a copy of the same object with only that single
09:50property affected.
09:52And that is what lens allows us here to do.
09:54We just do a set of the transmission, and the list of the messages that we want to assign
10:02it to, and then we get a new transmission with new results.
10:06So if we go over here, and then we just let the code go, we should now be having two different
10:12transmissions.
10:13A modified one, which contains only the phantom purpose in the phantom message, and we should
10:23be having the normal one, which is now, perhaps I could do this, I'm not entirely sure, no,
10:34we cannot do that again, yeah, we've, we lost it here, because, yeah, because we've got a
10:45coroutine there.
10:46And of course, when we're trying to debug coroutines, this happens, we lose the information
10:53about the previous run.
10:55And therefore, we don't have the transmission here at this point, we would have had if we
10:58would have called it.
10:59But that's not important.
11:00What's important here is that we are returning a modified version of this.
11:05And therefore, what we've got over here is the new message with that kind of purpose.
11:10Now, let's just check if it is the same message or not, by the way, if it is the same transmission.
11:14So if we go here to the modified, we can see that we've got sender, you, me, ID is null.
11:21And we've got this timestamp, which is exactly what we've got here, you, me, and the timestamp,
11:26well, the timestamp is the current timestamp that is used when we create the object.
11:30So we cannot check that, but we can definitely check that the sender and the receiver are
11:34the same.
11:37So now if I let this run, then that should mean that in the requests, yeah, there's a
11:47timeout, unfortunately, let's me just run this again, if I just let it go right now,
11:52we should now be able to see that we've got a response.
11:57And that response is the phantom purpose and phantom message.
12:01So this makes optics a very powerful tool.
12:04If you are thinking about changing data within an immutable, deeply nested data structure and
12:10want to give back that modified, copied version of the original one.
12:15All right.
12:16So this was optics.
12:18In our slides, we can now look at resilient communications with Aero Resilience.
12:24But what is Saga and what can do for us?
12:29Let me tell you about Saga.
12:31Managing long entangled workflows can feel like a chaotic game of Jenga.
12:36Because Saga is a transactional pattern that deals with different moving parts of a full,
12:42complete, distributed transactional system, it is our superhero to handle these transactions
12:50that may not have a lot to do with each other.
12:52Where we can, in each single point of action of these transactions, implement a rollback
12:58procedure, if anything fails.
13:01And that means that those rollbacks can occur successfully on the first time that an error
13:06occurs in any of the single steps of a Saga transactional pattern.
13:11With Saga, the workflows are organized, atomic, and recoverable.
13:16Because of the rollback I mentioned before.
13:19Saga can be seen as a fail-safe transactional system with a back pocket parachute.
13:24The thing about Saga is that there are two different ways of implementing Saga.
13:28One is an orchestrated way, where you've got one single Saga orchestrator dealing with
13:32all the different sub-transactions or actions of a Saga pattern.
13:37And you've got the choreographed pattern, where each one of them is responsible to give hand
13:42to the other next Saga action.
13:46In Project Arrow, we do the orchestrated one.
13:50Back to our slides.
13:52What's important here is that we understand that Saga is also an implementation, and it's
13:57also code.
13:59So let's see.
14:00The first thing that we want to understand about Saga in Project Arrow is that there
14:03is a Saga Builder, and the Saga Builder creates our Saga pattern.
14:08And we can have multiple different actions inside Saga, which are these ones over here,
14:13which are these Saga actions.
14:16In each one of these Saga actions, we've got two different parameters.
14:19One is the build body of a function that we want to execute.
14:24And the other one is the body of a rollback function to make sure that we go back if something
14:30happens.
14:31And the way this works is, when one of these Saga actions fails, the other ones will also
14:36fail back.
14:38So that means it will go up to one point, and then we'll do a rollback if at that point
14:42an error occurs.
14:43The only point in time where the whole transaction occurs is when all of these actions are successful.
14:51And that way, we can guarantee that we can get back to the initial state.
14:54So we need to be careful when we implement the successful action, and when we also implement
14:59the rollback action, which will give us back the system in its initial state as best as
15:05possible that we can do.
15:07Because there's no guarantee also that the rollback is correctly implemented.
15:11That is really up to us to implement it in the correct way.
15:15Finally, to run the Saga, and turning it into a suspend effect, we need to just call this
15:22Transact here.
15:24In this example, I am using Saga under an EIDER, and the reason for that is that I also
15:29want to handle the error.
15:30So here we see a good combination between using EIDER and using Saga at the same time,
15:35so that we can get either an app error, or the fleet user in this case.
15:40Because what we are doing here is, in multiple steps, getting the user to be filled out with
15:45the necessary information that it needs.
15:47The only thing is that, in each of the steps, we are checking if the user has permission
15:51to get it.
15:52So for example, in the first Saga action, it tries to get clearance to get a telephone.
15:57If that's grounded, it continues.
15:58If not, it will roll back everything that it has done before.
16:02And the same thing goes to the bank account number, and the same thing goes to the department.
16:06It's important that we understand that with a multi-step transaction, Saga is ideal for
16:11us.
16:12So every time we think about a multi-step transaction, if it is distributed or not, we will always
16:16benefit from using this pattern, and Project Arrow offers this right out of the box.
16:23But let's have a look at an example and see how this works.
16:26So if we go to our code and look for Saga, and we go over here, we find the exact same
16:34method that we have just seen in our slides.
16:37This register user by ID.
16:39And what it does here is that it essentially is ensuring things, it's making sure that it
16:45is using this raise inside this idr.
16:49It's trying to ensure that original user is not known, for example, and raising and not
16:54found.
16:55And it's making all of these different checks.
16:58And the important bit is that, for example, in this Saga is the builder.
17:02And then here we've got one Saga action that will perform this operation if it succeeds.
17:06And if it doesn't succeed, it will then run this piece of code, which will then do a rollback.
17:12This is the rollback.
17:13In this case, I'm applying a nullable, just in case that the original user is null, then
17:19it will fail, but it will not fail the rollback, it will just go back, and it will silently just
17:25ignore the fact that the user is null.
17:30And the same thing goes for all of the others.
17:32Now, at this point, we should be able to understand already what this nullable does.
17:37So essentially, it returns null if any of the properties that are going to be bound
17:42are also null.
17:46So in this case, if original user is null, it will short circuit the function, and then
17:50it will return null to here, which gets ignored, and then nothing happens.
17:53And then it can proceed with further rollbacking.
17:58So let's now see this in real time.
18:00If I call this user's ID register, we should then be able to affect the current user and make
18:10sure that the user gets updated with the data that we need.
18:13We also got here at the top a GET that will give us the current user.
18:19So let's see which user do we have there at the moment.
18:22Now, this is not using a database in the back.
18:24It's just using a map.
18:25And that is specifically made so that we can simplify the way we see Saga implemented in
18:31our code.
18:32And if we now go to another file, which is called Saga requests, we will find the different
18:35examples that we want to use.
18:37The first example, I see that it is user 1.
18:40And I know this for sure that we've got a user 1 that will simply return, I believe, me,
18:47is that me?
18:48That is me.
18:50At example.com with no telephone and no bank account number and no department and no chamber.
18:56Okay.
18:57Good enough.
18:58So now let's see what happens if we now try to send a request, where I will try to put
19:04all of this in the code.
19:06So now we know that this is going to go via Saga.
19:08So if I run this, then what happens is that if I try now to get the user back, then the
19:18user has got everything filled in.
19:21That means that the user has got permission to get all of its data filled in with the telephone,
19:28the bank account number, the department and the chamber.
19:31Now let's go back to our Saga and let's purposely create a problem there.
19:38Let's just say that right at the very end, we are going to simply create a real big problem.
19:46Let's say that the user didn't got any clearance.
19:48And let's just say, let's just throw here some exceptions and runtime exception.
19:54And if this runs, then that means that the last step is wrong.
19:59So because this is all in memory, I can restart the service and make sure that the user gets
20:03back to having null values.
20:05Let's see if that is the case.
20:06So if I go back here to Saga and simply run this, then I see that the user is back to
20:14having null values in the telephone, bank account number, department and chamber.
20:18But now I am forcing a problem to occur at the last stop.
20:22And that means that when I run this, I will get a 500 internal error.
20:27But if I now try to run this get, the user hasn't changed because the rollback has occurred.
20:34And the rollback for all of the different actions has occurred as that means I do not
20:38have a chamber.
20:39I do not have a department.
20:40I do not have a bank account number or a telephone because the user was updated up until a point.
20:45And then it got rollback from that point all the way to the beginning.
20:50But again, the responsibility to create a rollback is our own.
20:54All right, so this was the Saga pattern.
20:59Let's now have a look at something different.
21:01This one is a very simple thing that revealed to be very interesting and very useful in our
21:06project.
21:07This one is called a synchronous data streaming with error effects.
21:11Here's the thing, in all of this code, can you identify where the good use has been
21:18made with error effects?
21:23If you point it out to using this parzip, then you're right.
21:27That is what we are going to talk about in this very short bit of talking about the error
21:33project.
21:34So parzip is interesting, and let me tell you a bit about parzip.
21:37If launching coroutines in parallel is something that you are interested in, another thing
21:42that you may be interested in is that your coroutines come back after running, and like
21:46a well-trained squad, they come back all together for the perfect photo.
21:50And you are interested in that, then parzip can be something for you, because it will
21:55wait for all the coroutines to come back together and get those results of all the different
22:00coroutines executions, pick them up, and make something new that you might be interested
22:06in.
22:07So parzip makes sure that the execution of different coroutines can be run in parallel, and we
22:12can get the results in the end after the runtime execution.
22:15So that means there are no early exits, no one first back wins, and it's kind of like
22:20a brunch where everyone needs to be seated first before we start our meal.
22:26And that is the power of parzip.
22:28So let's go back to the slides and see this in more detail.
22:33So parzip works in this way.
22:35There are two different functions, two different bodies of functions.
22:38One is the transmission service get transmissions by package ID, and the other one is message
22:42service get message package by ID.
22:45These two will run in parallel, and then their results will be picked up at the end with the
22:51sender timestamp function that receives these two parameters, and then makes a sender message
22:56detail with the sender and the timestamp.
22:59Let's have a look at this in our code.
23:01So if we go to our intelligent right now, and if we look for parzip, parzip, we will now
23:09find that we've got this endpoint called purposes, and purposes, it will essentially create a sender
23:16message detail entity.
23:18Let's have a look at what it is.
23:20It is a DTO that has a sender and has a package timestamp.
23:24What's important here is that we can fill this out in any way we want.
23:28We will just put first the sender, and then the package timestamp with the local date time.
23:33And of course, we can see here the serializable with the manual serializable.
23:37We have annotated the class with serializable, and this is important.
23:42One thing that is important to know that I didn't mention yet during this presentation
23:44is that serializable in this case is not really mandatory.
23:47It's just that I'm using Jackson, and therefore, I have to use the annotation serializable
23:52here.
23:53It doesn't have to be this way.
23:54We can use other different libraries, but that's the one I chose for this example.
23:58So serializable here is not really related to the topic of this video, or even with ktor
24:03itself.
24:05So let's now see.
24:06We are calling parsip.
24:08And parsip, if we go to the definition, parsip has many different ways of being defined.
24:14We've got here AB with another parsip, but we also have ABC, for example, we've got ABCD.
24:23So we do have different kinds of implementations of parsip, different overloading methods that
24:28will just use different function bodies that will run in separate coroutines.
24:32In this case, we can see that it runs via the coroutine scope with its default coroutine
24:39context, with the one that where they are in at the moment, and then we will get the combination
24:44of all of them into a type called E. And that E type is the type that we define inside
24:49the body of the function that collects all of the data of the different coroutines.
24:54So if we go back now, we will see that we will get the sender and the timestamp.
24:58But how do we get the sender?
24:59The sender we get here on the first function, which is by using this particular get transmission
25:05by package, and it will then get the sender from the package that has a certain ID.
25:12Now let's have a look here first at this thing specifically with bind-bind.
25:16So we've got three different binds here that will shortcut potentially our function, because
25:23we are calling here nullable, and nullable here will send a message, will signal the contents
25:31of this to return null when one of these properties is null.
25:35So if the package ID is null, then everything will be null.
25:39If the package ID is null here, then everything will be null.
25:42It means that the result will be null in this case.
25:45So for example here, if there is a null point here, then that means that this will all return
25:50null and we will get the timestamp with a null value.
25:53In here, the same thing, if we get package ID to be null, then we will get a null value
25:58for sender.
25:59But we will also get a null if the result of the get transmission by package ID is also
26:07null.
26:08So if we get no transmission here, we will not be able to get a sender.
26:11But this is covered here with bind, because bind will shortcut this function, which will
26:14then return a null, and we will get a sender message details with a null value that will be
26:19sent back to the client with a null value.
26:21So that's not what's important here.
26:23What's important here is that we are using Parasip to launch two different coroutines,
26:27one that launches this function, and the other one that launches this function body.
26:30We've got two different function bodies that will be run in a coroutine context that will
26:35then be collected at the end to create a sender message detail object.
26:40So let's now see how this works in our code.
26:42So let's find the purposes example in requests.http.
26:48So if I go here and try to find it, I can see here a get.
26:53And if I use the get right now, I should now get, well, I get nothing, because it's empty.
27:00So if I go here to routing and try to get the purposes, then the get purposes will try to
27:11get all the messages from the service.
27:13And therefore, this package ID that we would get from the messages is not existent.
27:19So there is no message, so that means that the sender message details will just be an empty
27:24object in this case.
27:25So it will be an empty list.
27:33Let me just correct one thing here, which is I can simply map not null.
27:43So that way we will not get a null value in any of the responses.
27:47So right now, the way this can work is if we first add some messages.
27:51So we can just go back to our example and just simply send one transmission.
27:56So if we go here all the way up and up to, let me just go back here and perhaps it's also
28:08interesting for us to roll this back.
28:12And then here we can then send this transmission.
28:16And now we are putting this transmission in our database.
28:22This is actually a real database because it's going via expose.
28:26And so now, if we now run the same example in our requests, this get, that means that we
28:41are going to get one value.
28:43And that value is sender u with the package timestamp, because that's what we wanted to
28:47achieve.
28:48We wanted to achieve getting the sender of all of the different messages.
28:52We have sent one single message, and then we want the package timestamp.
28:55But the package timestamp, if we go back to our results here, we get the message timestamp
29:00by simply getting the message package by ID.
29:05So every message gets a package ID assigned to it, and with it we can find the package timestamp.
29:12So that's how this whole thing works, and that is how this parsip can be useful for us.
29:18And let's not forget that parsip has different kinds of overloads, so that we can add more
29:23function bodies if we actually need them.
29:25So if we need two, three, four, more function bodies, there are functions that cover that
29:30in our arrow library.
29:31All right, so this is the part of parsip.
29:36Now let's have a look at something that is interesting, that I have actually already made
29:41a video about it.
29:42That is this one.
29:43I encourage you to go and watch it, but only after we have a quick look at the slides.
29:48So let's go to our slides.
29:51So that is about consistent state management with arrow STM.
29:57This is the overview.
29:59And the way this works with arrow STM is that we are trying to cover an example of keeping
30:07state in memory and managing state without corrupting it.
30:11So we want to make sure that our state is managed in a transactional way.
30:15This is different than accessing databases.
30:18It could potentially be used in combination with databases with the different isolation
30:23levels that the database has to offer.
30:25But in this case, the most important bit is that we can manage state without a few of the
30:31inconveniences of normal locks.
30:34So what this does is we avoid problems with race conditions and with locks.
30:39So that means that we avoid locking our database and we avoid creating a deadline.
30:45How does this work and how does this work, particularly in memory management?
30:51So here we've got two different transactions, transaction A and transaction B. Transaction
30:55A is doing something and transaction B is trying to do something.
30:58But what happens is that in this particular case, both of them are successful and there's
31:03no particular problem.
31:04So this is the typical case.
31:06We do a read, we do a modified data, we commit.
31:09In both cases, everything is successful and everything runs successfully.
31:12But this is the easy bit.
31:14When it gets complicated, it's when we get across the first problem, the first, not even
31:20an error.
31:21It's more like the first conflict.
31:23If transaction A gets to modify data and transaction B modifies the data, by the time transaction
31:30A commits, it will be successful.
31:32But by the time transaction B commits a change to the same transactional variable, it will fail
31:39because it attacks a conflict.
31:40But the great thing is that it will retry implicitly one more time to make sure and see if the transaction
31:48is successful or not.
31:49So basically, STM is based on retries and not based on locking and unlocking or releasing
31:55state or getting state or getting access or not getting access.
31:59It's just basically a retry mechanism that will manage the state in memory.
32:05And that means that we don't have the concurrency issue, we don't have race conditions, and we
32:11don't have any issue with deadlocks.
32:14In a more complicated case, we can see that we can get transaction A to modify data and
32:21then transaction B will try to modify the data as well.
32:25It gets a conflict, it retries, but somehow, then there could be another error, there could
32:30be another retry, there could be another transaction that is going to modify the same transactional
32:36value in memory.
32:37And that means it will keep on retrying until it's successful.
32:40There is never a period where it keeps retrying forever because that's just not possible.
32:44It will retry until it achieves to commit or until there is an error.
32:51So there's no retries if there's no transaction occurring, changing the state of a particular
32:56transactional variable.
32:59There is only a retry if that happens.
33:03But of course, at some point, all of them will have retried.
33:06So at some point, there will be none of them busy with retrying because the state has been
33:11changed.
33:13And they will fail if there is an actual failure.
33:17So there will be no retries on an actual fail.
33:20Not on a state change, but on an actual fail.
33:24But how does this occur?
33:25Well, here's the data model that I've used.
33:26It's a very simple one.
33:28There's just two different types.
33:29One is docking bay and the other one is fuel station.
33:32And these ones will change when we apply TVAR on them.
33:37So docking bay and fuel station on their own, they are not transactional variables.
33:41They are just simple, simple types, simple variable types.
33:47But when we apply TVAR on them and initialize them that way, then they become transactional.
33:53And we are able to use them in a transactional software memory way.
33:59So let's see what happens now.
34:02But let me tell you about software transactional memory.
34:05Have you ever wished that your threads would cooperate rather than fighting for resources?
34:10STM in Arrow allows to do just that.
34:12With STM, you can run all different concurrent operations as if they were part of one big happy
34:17family.
34:18STM in Arrow brings you the power of transactional memory where actions are atomic, isolated, inconsistent.
34:25So back to our slides.
34:27Talking about consistent state management with Arrow STM means that we also need to understand
34:32our code.
34:33And for that, here are the examples that we are going to have a look at in our slides.
34:38Remember, the video is available.
34:40And please have a look at it because it explains all of these examples in detail.
34:45So I'm not going to repeat the practical case of these examples.
34:48So consistent state management in this case.
34:51What we've got here is a late init var called docking bay and another late init var called
34:57fuel station.
34:58These are all declared as transactional memory variables that are initialized by using this
35:04initialized function where we can start them with certain values.
35:11The idea here is that we are calling this initialized from another way of creating services
35:16in Ktor.
35:17And that is using coin.
35:20And coin is just another library that allows us to use injection in Ktor.
35:24It's not really important for the example.
35:26But the idea that I want to give you is that I created these variables somewhere in memory
35:30where they are now transactional.
35:32And that means we are only be able to use these transactional variables with the use of
35:39things like atomically or STM, which we will see right now.
35:43So to run a transaction to completion, we use this atomically function that will then receive
35:48a body that we will define.
35:50And then we are able to process it.
35:52The way it works is that since we are using docking bay as a transactional memory variable,
35:58that means that it has different functions and methods to be able to change its value.
36:03One of them to read is called read and the other one to write is called write.
36:08And there we see our docking bay right over there, where we read the value and if there
36:12is enough space in the docking station, then the ship can park.
36:16And that means that the ship will occupy that, meaning that the state of that value will be
36:22changed after committing.
36:23And committing means reaching the end of atomically.
36:29In the same way, we can break apart one single transaction with the use of STM.
36:34STM can be placed inside of atomically so that we can separately create different separate
36:40transactions so that we can perform changes that we want to achieve.
36:46For example, here in the fuel station, we are trying to fuel our spaceship independently,
36:51so that we can keep control of how much fuel is still available and how much fuel the ship
36:58is allowed to take.
37:04And with STM inside atomically, we also are able to provide a fallback.
37:09That means if something fails, what do we do afterwards?
37:12So for example, if fueling fails, what can we do?
37:14Should we use another fuel station?
37:17Should we try to use the resources?
37:19Should we try to use some kind of storage that we have inside our own spaceship or use
37:24some kind of alternative green energy, for example, in our spaceship?
37:28These are alternative things that we can do should fueling fail in this case.
37:33So it is very complete.
37:34We can use atomically with an STM which provides more extra functionality in not only just doing
37:40different transactions inside another transaction.
37:45Another thing is when we are talking about failures.
37:48When we are talking about fail, in the example that I gave in the other video, is that if there
37:55is a fail, then the transaction stops.
37:58And that's what we see over here, if the requested fuel is 20.
38:01This is just for tests, so if I put a 20 to the fuel, that means that it will immediately
38:06throw this illegal state exception, meaning that the transaction will stop, there will be
38:11no more retries, and we will finish the transaction with unfortunately a failure.
38:18And that's what this atomically does.
38:23So this means that by throwing this error, the transaction will stop, and nothing will
38:28be changed.
38:30Now, as a conclusion, which is important now to get everything together.
38:36Saga.
38:37What is Saga?
38:38It enables a robust transactional workflows in distributed systems by handling compensations
38:42and rollbacks.
38:43It is true, we saw in the example that we can perform different operations where we need
38:47to validate each one of the different operations, and if anything fails, it will fail back to
38:52the initial point.
38:53We will be able to reset to the start point of our transaction.
38:58The only thing is that we need to be very careful so that the rollbacks are working correctly,
39:03so that we can get back to the initial state.
39:07Then we've got Nullable.
39:08And Nullable, we have seen that we can use this with Option or with null values directly.
39:16If we want to use null values directly, we can just use the Nullable function.
39:21If not, we use Option, so that we can use Option, which has then the two different subtypes
39:26that we have seen, the None or the Sum.
39:33We've got EIDR, which allows us to deal with errors in a way where both the errors and the
39:37results that we need to get are held together and are accommodated in our EIDR object so that
39:45we can deal with them and perform safe operations in a functional way.
39:51We have also seen Optics, which provide us, of course, a way to deal with deeply nested data
39:56structures that are important for our system, and we, that way, don't have to generate so
40:02much complicated code.
40:04Which, of course, is always the goal when we are implementing code and we are implementing
40:09services.
40:10We don't want our code to be too extended, and we don't want to have too much complexity,
40:14and, of course, we want to make our code as simple as possible.
40:17Which is something that Optics provides very well.
40:19The only disadvantage, perhaps, is that we need to use plugins, and we need to choose
40:24a specific kind of plugin that can process symbols, and we also need to use the library
40:29that Optics provides, because otherwise it will not be able to interpret our classes.
40:35And of course, it relies on code generation, which can be a hassle when we are generating
40:40our bytecode for our services to work.
40:45We've got zip or accumulate, which is a great thing in combination to use with either.
40:50We can accumulate all of the different validation problems and put them in a single list that
40:55we can then choose to be either a string, which is the most simple case, and I do not recommend
40:59using string, but we can use exceptions for it, and we can put all of the different exceptions
41:04that we have got, but when trying to access a function, put them there, and then we can process
41:09each one individually.
41:10Maybe we want to give them back to the client, or we just want to process them and register them
41:14in a database, send them to a monitoring framework, whatever we want to do with it.
41:19It is a very interesting thing to use.
41:23Finally, STM.
41:25STM provides safe composable concurrency without logs, without race conditions.
41:30STM allows us to deal with all of the different data that is held in a state, that is held in
41:38memory, and allows us to do transaction management without the need for isolation levels, or any
41:45kind of optimistic or pessimistic locking.
41:47It is just a mechanism that is based on retries, automated retries, and we see that implicitly
41:54in our code.
41:55Again, this video explains very well the different steps, and I explained there how the example
42:01works, and where can we see the retries happening, and how can we see the retries happening, although
42:07that's not something that it's necessary to do, but I did it for the example so that we
42:10understand exactly how STM works.
42:13All of this is something that we can see in our projects, it's something that we can use
42:17in our projects.
42:19These are things that we found out to be the most useful for everything that we are doing.
42:25I think this is something that can be a generic template to use in many different projects.
42:31We can have an approach of trying to use everything that Project Arrow provides, which is, I think,
42:36an unattainable goal, because not all of what Project Arrow provides is something to use in
42:41a generic way.
42:42There are niche potentials there, and we need to make sure that we do have those niches in
42:48our project.
42:49If we want to use those interesting things like, for example, prisms, which is a very interesting
42:53topic of Project Arrow, but of course, it doesn't fit the general usage that Project Arrow can
42:59have for our production environments.
43:04Questions.
43:06I had a ton of questions during the presentation in London.
43:11I had three particular questions, and they were all very interesting.
43:16There was one question about EIDA being deprecated in Scala.
43:20This is something that I need to investigate further in another video.
43:23I wasn't expecting that question.
43:26EIDA is very much used in Project Arrow, and we use that a lot.
43:31And we see a lot of advantages in using EIDA.
43:34So that is an important, interesting aspect.
43:39Perhaps Scala doesn't want to use EIDA anymore.
43:41I'm not entirely sure about that one.
43:43So it's a good topic for investigation.
43:46The other question was about debugging.
43:47If we could debug things with nullable option or either.
43:50Yes, we can.
43:51We have seen in these examples that we can basically debug everything, and we can see the results
43:59going in.
44:00The only thing is that we are always facing that problem with Kotlin coroutines.
44:03If we are having coroutines in our code, then we lose the debug points from the coroutines
44:09if they are, if we are past the point of the coroutines running, and we don't hold a reference
44:16to those variables.
44:18And finally, there was a third question about STM, about software transactional memory.
44:23It was more about how the hell does software transactional memory doesn't get locked?
44:33How deadlocks do not happen?
44:35Well, again, the video where I talk exclusively about STM explains that in detail.
44:41But in general terms, because it is a system based on retries, locks will never happen.
44:46And I also explained this now in more detail, because I wanted to focus on that question
44:51and make sure that that question would be very clear what the answer would be for it.
44:56And of course, STM, because of the retry mechanism, there will never be a lock, there
45:02will never be a transaction waiting for the other, because one will always finish.
45:07And if it doesn't finish, it retries, it retries, it retries until all of them have finished.
45:11And then the only possibility that one of those transactions will not succeed is if at that
45:17point, an error occurs.
45:20Otherwise it will be successful and the commit will be successful.
45:27These are the resources for this project.
45:30It's called Spaceship Adventures.
45:32It's available on GitHub.
45:34It's available under my handler, j.espanensignan.
45:39These are the resources that you can check out to learn more about the meetup, about Project
45:44Arrow, about Kotlin, and of course, my own website.
45:50Then about me.
45:51So I have this homepage, which is juanfilippes.espanensignan.nl.
45:57It's my full name.
45:58LinkedIn.
45:59You've got YouTube, BlueSky, Mastodon, GitHub, Hackernoon, and DevTO.
46:03So this is where you can find me.
46:05If you want to pop in a message about this presentation, I really appreciate that.
46:09And then for the last bit, I just want to thank you, the viewer, for watching this video.
46:16Congratulations for reaching this point.
46:17I want to thank the London Kotlin community for giving me the opportunity to speak there
46:22at the meetup.
46:23I found it a very different way, an interesting way of giving a presentation because it was
46:29done online.
46:30We have photos of that and a small video about the questions that were being made.
46:35I found it to be a very interesting and rewarding experience.
46:41And I learned a lot from it.
46:44And one of the things that I've learned is how to speak to the silence because there were
46:53people listening to me, but I couldn't hear anything of what they were saying.
46:56So I was literally feeling like I was talking to a void where I really wasn't.
47:00And there were lots of people listening to me.
47:02But the silence was so extreme that it was difficult because I really liked the interaction
47:08with people.
47:09But it went really well.
47:10In the end, actually, everyone really heard really well what I was saying.
47:14And the microphone worked really great.
47:16And all the connections worked really fantastically.
47:18But again, it's a different experience just speaking to a screen and speaking to the slides
47:24and thinking that people are listening to you.
47:26And they were because the questions were great.
47:28And people were very enthusiastic about the presentation that I gave there.
47:32So that was really nice.
47:35So yeah, I really want to thank the Kotlin London community for this.
47:41I want to give a special thanks to David Kraft for keeping contact with me.
47:48We actually contacted via Blue Sky.
47:51And then all of a sudden, we were talking about this in Slack.
47:54And then it finally resulted into having this presentation being done online in the London
48:01Kotlin community.
48:02It was a bit of a shame that we didn't get around to get everything in order and correctly
48:09made so that I could travel to London.
48:12But hey, no problem.
48:13It was still great.
48:14It was still amazing and really nice to be able to give this presentation online for the London
48:21community.
48:23All the details of this meetup are available in the description.
48:26Make sure to read it because I also put the links to anything that is relevant for this
48:29presentation right over there, including the link to the STM video with Ktor and the other
48:36Ktor video that I made so that we kind of understand all of this stuff that I've discussed in this
48:42video, especially if you are coming from a different kind of world, like the spring world, and you
48:48don't know much about Ktor.
48:49It is important also to have a look at this video over here where I talk about the basics
48:53around Ktor and why is it important for us to understand and learn more about a different
48:58way of creating services.
49:01And make sure to give us a like.
49:03It helps a lot if you give us a like.
49:05If you give us a bit of a comment in the comment section and make sure to subscribe if you enjoyed
49:13this video because I have a ton of videos and a ton of ideas to put in this channel.
49:17This channel is always moving.
49:18There are more busier moments than others, but that's life and sometimes there is not
49:24much time to make videos.
49:26In the coming period, there might not be too many videos being posted on the channel, unfortunately,
49:31and that is because there is going to be a very busy time for me in the coming weeks.
49:36And until the next video, be sure to stay tech, keep programming, be kind, and have a good
49:45one.
49:46Bye!
49:48Bye!
49:49Bye!
49:50Bye!
49:51Bye!
49:52Bye!
49:53Bye!
49:54Bye!
49:55Bye!
49:56Bye!
49:57Bye!
49:58Bye!
49:59Bye!
50:00Bye!
50:01Bye!
50:02Bye!
50:03Bye!
50:04Bye!
50:05Bye!
50:06Bye!
50:07Bye!
50:08Bye!
50:09Bye!
50:10Bye!
50:11Bye!
50:12Bye!
50:13Bye!
50:14Bye!
50:15Bye!
50:16Bye!
50:17Bye!
50:18I've updated my disclaimer to better reflect my position.
50:29This video is not sponsored and I have not received financial compensation for its content.
50:35Any brands, products or companies mentioned are for informational purposes only.
50:40While I may participate in various programs, meetups and conferences, all opinions expressed here are my own.
50:48For more information visit www.fema.org