Proto.Actor based API with OpenTracing monitoring

Actor model has been around for quite some time, and modern implementations like Akka and Orleans made the model even more popular. I wanted to try out the actor model for the first time and for the exercise have chosen a light-weight and little bit less known implementation — Proto.Actor. In this blog post I will go through a naive RESTful API based on Proto.Actor and will leverage it’s Jaeger OpenTracing plugin for monitoring.

TL;DR;

The source code of this Proto.Actor exercise is on my GitHub, and readme file contains instructions how to run the solution.

Proto.Actor

Proto.Actor framework has been developed by one of the creators of Akka, and the main selling points are lightweight and performant implementation of the actor model. If you checked the number of lines of the code of Proto.Actor (GitHub), then you would see how light the implementation is :) I’ve also found the benchmark example there, as well dozens of other useful examples.

The exercise

I’ve built a naive User Management API, which allows to retrieve all existing users, retrieve single user by ID, create a new user with a name, and delete a user. That’s it!

There are following projects in the solution:

• Actors – actors and their behaviour
• Api – API endpoints, controllers
• Commands – models used for in-memory commands
• Domain – the main business logic
• Events – the events raised by domain
• Persistence – in-memory persistence implementation
• Domain.Tests – unit tests for domain logic

The API’s logic is super simple, however just to indicate the boundaries, all the business logic is separated in Domain project. Having that said, the whole exercise is more about tooling, specifically Proto.Actor, OpenTracing with Jaeger, and as a bonus — SEQ logging.

Running the API

To launch only the API (in case you don’t have Docker installed) it’s enough just to run the API project from command line with “dotnet run” or from your IDE in debug mode, however to fully follow the exercise and have Jaeger and SEQ running, you will need to have docker installed on your machine (Windows Home doesn’t support HyperV, hence can’t run Docker!).

To run the whole solution in Docker run following command:

docker-compose up --build

Now you can hit the following endpoints using a REST client like Postman (see this blog post for more about Postman):

GET     /users
GET     /users/{id}
POST    /users
DELETE  /users/{id} 

Actors setup

There are two actors in the API — the parent routing actor, and child user actor. The parent is intended just to do the routing from controller to the actual user-related operation, and delegate the work to the child actor. The parent is not expected to be affected by any infrastructure i.e. persistence, and should not be crashing often, hence selected as the parent.

From the available Proto.Actor plugins, I’m using actors factory in Startup class to setup both actors, see snippet below.

services.AddProtoActor(props =>
{
	props.RegisterProps<RequestActor>(p => p.WithChildSupervisorStrategy(new AlwaysRestartStrategy()).WithOpenTracing());
	props.RegisterProps<UserActor>(p => p.WithOpenTracing());
});

And inside RequestActor class the parent is calling the child actor, and requesting a Future to be returned when ready. I am returning a domain event, which then is returned to controller.

var userActor = _actorManager.GetChildActor(ActorNames.UserActor, context);
var userEvent = await context.RequestAsync<UserEvent>(userActor, message, _childActorTimeout);
context.Respond(userEvent);

And inside the ActorManager I am relying on IActorFactory to get actors for me by the ID.

Domain model

A child actor can perform following operations that match API endpoints:

• Get all users (command which returns query result)
• Get single user (command which returns query result)
• Create user (command)
• Delete user (command)

All this logic is in Users class which is my domain model, and manages the users list. The list of the users have to be injected into the domain model to support restoring the object state using snapshots.

Also, each operation must be able to re-process previous events, therefore there is no strict parameters validation. This is not a super protected domain model and its state can be effectively modified by an outside action (injected via constructor), however this way I’ve easily enabled Event Sourcing support.

Since actors are event-driven by nature, Event Sourcing feels very natural when working with the actor model.

OpenTracing setup

OpenTracing tries to solve monitoring problem by correlating all the involved services. In my sample API, we can even track requests traveling between actors within the same process.

Installing two nuget packages allows to start using Jaeger right away:

• Proto.OpenTracing
• Jaeger

I am using appsettings.json to configure Jaeger.

  "Jaeger": {
    "JAEGER_SERVICE_NAME": "UserManagementApi",
    "JAEGER_AGENT_HOST": "localhost",
    "JAEGER_AGENT_PORT": "6831",
    "JAEGER_SAMPLER_TYPE": "const",
    "JAEGER_SAMPLER_PARAM": "1",
    "JAEGER_SAMPLER_MANAGER_HOST_PORT":"5778"
  }

Then in Startup class it’s just two lines:

// ... ConfigureServices()  
services.AddSingleton(Jaeger.Configuration.FromIConfiguration(_loggerFactory, _configuration.GetSection("Jaeger")).GetTracer());
// ... Configure()
GlobalTracer.Register(tracer);

If Jaeger is not running, the plugin will simply ignore all the monitoring events sent to it. And if the setup was done correctly, you should see the API registered with the Jaeger. Make one API request to test the tracing.

One caveat here — I use “win.local” localhost domain alias pointing to my machine’s external IP address for local develop so my application can connect to docker containers and vise-versa. In docker-compose.yml file, however I use docker container names instead of domain names.

And just try out BaggageItem feature, I’ve added a very simple chaos engineering feature to fail API request when creating a user, see the code snippet below, or the full ChaosMiddleware in the source code.

if (chaosTypeValue == Headers.CreateUserDown)
{
	var span = _tracer.ActiveSpan;
	if (span != null)
	{
		span.SetBaggageItem(Headers.ChaosType, chaosTypeValue);
		_logger.LogWarning($"Attempting to enable Chaos Engineering mode with type '{chaosTypeValue}'");
	}
}

You can turn on and off the chaos engineering via HTTP header in the request:

chaos_type = create_user_down

Jaeger and SEQ docker setup

I’ve included docker-compose.yml file containing all the necessary services to be run in docker with a single docker-compose up command.

Together with the Jaeger image jaegertracing/all-in-one:latest image I had to expose all the ports Jaeger is using, and also override the local config agent host value, so Jaeger can resolve it’s instance.

- Jaeger:JAEGER_AGENT_HOST=users-jaegertracing

It’s much easier to setup SEQ than Jager — just run datalust/seq:latest image, and expose 5341 port. The data volume setup was skipped intentionally to have temporary storage for this specific project only.

Considerations

The User Management API needs to await for the parent Future to return a child Future value, however it would be more efficient just to keep processing messages between actors. It now feels somewhat like using MediatR :)

Actors are not persistent, therefore if the app crashed, or container was restarted, then message in flight would be lost and never retried.

Jaeger provides UI, however as a tool for monitoring I would expect some dashboards, or more flexibility when query/grouping data just to be able to do the actual monitoring. Having that said, there seem to be some Grafana plugins to try out for the next time :)

Conclusions

It was a fun exercise to try out the actor model, a slightly different mindset. And the Proto.Actor implementation had many plugins to work with that made my life easier. Also it felt very natural to work with Event Sourcing when using actors.

Jaeger setup was easy and it provides very nice .NET SDK for integration, however if you are tracing requests between remote services and not using HTTP for the communication, you might need to implement your own requests correlation logic following OpenTracing standards. And Jaeger UI was quite limited.