Introduction

Lately I heard about Typesafe's Jev system-one model, a new AI model that's not intended to generate output as text, but rather to evaluate input text and return a structured decision with probabilities. The input contains relevant information, define questions and possible answers, then the evaluation results are returned

I wanted to test out these decision models with something I'm familiar with: AWS. So I built an incident-triage workflow that uses close-to-real AWS logs from different sources in order to classify an incident in a category and whether the incident required additional attention or not

I tested each model against some synthetic logs. Each model's run was evaluated on accuracy of the classification, if it was correct on the "attention required or not" and how log it took

The most interesting thing to me with this experiment is checking whether smaller local models could handle the task well enough to be useful and compare the time they took compared with the API model (luna6-decisions in this case)

What "decisions" models do

To a high-level, a "decisions" model receives a set of pre-defined possible answers and evaluates an input (in this case, text with the json logs from AWS). Then the model classifies the input into a set of probabilities of the pre-defined set of possible answers

Types of primitives and examples of use cases, answers and output. Taken from https://docs.typesafe.ai/concepts/system-one

The models tested

The comparison I wanted to make is between open-sourced models and api closed models. Since I use ollama for local model usage, I just checked through ollama which decision models were available. For the api closed models, since I'm more familiar with openai models, I went with luna6-decisions

It is important to note that variants and models that relied on MLX runtime were not tested and measured, as that was unavailable in windows and the experiments were run with Ollama for windows

The Experiment

The dataset that was created is close to how real AWS logs from different services would be found: AWS Lambda, DynamoDb, RDS, SQS and API Gateway issues were created as a list json with an id of the log (random) and the actual log.

These logs list where read from a node.js script (running locally in my computer) and fed into the decision models through ollama and openAI API.

The idea was to measure if the issue was classified in the appropriate category and if attention was needed for that issue or not. Also measure time it took. Those results were saved as files for later analysis

The accuracy responses of each model's responses was measured with a 50% probability cutover. Meaning that if the model's response had a category with probability higher than 0.5, then it was flagged as "correct"

The tests for open models were ran in my desktop machine, with one RTX 5060ti which has 16GB of VRAM, a Ryzen 7 processor and about 32GB of DDR4 RAM

High-level diagram of the experiment. Own Creation

Results

Here's a table that shows the results of each model evaluated - October 19th. Accuracy includes all the 139 log issues, including failures. Mean request time is calculated even taking into account the first request as well as the failures

Given these results, we can see that no model scored 100% on category accuracy. Not even the bigger models or the api closed model. Some smaller models gave a decent accuracy given their size, like d1-3b (84.9%) and even tev1-0.8b (73.4%)

In the "issue requires attention", clef 27b, pplx-decider-27b and luna scored perfect result, while others came very close. Particularly, tev1-4b did pretty well (96.4%) give its model size. The kev-4b also gave pretty good results for being a relatively small model (4b)

Conclusion

It's important to note that, the results and insights presented here are not to be interpreted as accuracy in all different contexts and solutions. The experiment was very specific to a use case and used synthetic data to evaluate the models, so it can't be interpreted as a production accuracy either.

The idea was to measure different open-models in a familiar setting to me (AWS log analysis) and see how they fared given different sizes and expectations

Some insights we can extract from the results are:

• Large models doesn't automatically mean better performance. Tev1-4b classified more incidents correctly than clef-27b, although clef made better attention decisions
• The highest accuracy came from pplx-decider, but took almost 9 times more time than the next better, gpt6-luna. The difference in accuracy wasn't all that notable to justify the extra time. Given my hardware limitations, it may be possible that pplx-decider running in a beefier machine would perform better latency-wise
• The results found suggest that self-hosted decisions models deserve to be considered for classification tasks. tev1-4b had a 96.4% attention accuracy with an average response time of 220ms, on a single consumer GPU (uses aprox 5-6GB). I think it's a strong option for developers to have a local option, and even for larger teams that want a cost-effective classification option or with self-hosted requirements

You can check the experiment and run your own tests. The following is the GitHub Link