We used two public datasets. GoEmotions is Reddit comments, 43,404 of them once its emotions were mapped onto our five labels. The Financial PhraseBank is 4,846 sentences from financial news. My job was to get both into one shape and hand the team data they could trust.
All cleaning and splitting lived in a single script, and the README carried one rule in bold: do not write data cleaning or splitting code inside your modelling notebooks. Everyone loaded the same files.
The finance sentences were split 70/15/15 into training, validation and test sets, with a fixed seed, keeping the label mix the same in each part. That gave a test set of 727 sentences that no model was meant to see during training.
Most of our models scored between 82% and 85% accuracy on that test set. Then the model trained on both datasets together came back at 98.9%.
A jump like that is a symptom, not a win. The cause was in the data. The combined training file had been built from the full finance dataset, all 4,846 rows, which included the 727 test sentences. The model was being marked on sentences it had already seen.
The fix was to rebuild the combined file from the Reddit comments plus the finance training set only, 46,796 rows, check that no test sentence was left in it, and retrain. The score fell to 84.7%, in line with everything else.
With the leak gone, the results were less exciting and more useful.
The stratified split matters for the same reason. Fear is under 1% of the finance data. A plain random split could easily put 4 fear sentences in the test set, or 10, and the comparison between models would change with the draw.
I did not build the models. My teammates did, and the modelling choices are theirs. What I owned was whether the numbers could be believed.
The habit I kept is that a split is something you verify, not something you assume. And the result you are happiest about is the first one to check.
The code and the dashboard are on GitHub, and there is a short summary on the work page.
]]>The app has short lessons, a guide to finding care, a bill explainer that works on made-up invoices, and a chat that answers general questions from the insurer’s member guide. My part was the bill explainer’s extraction, the benefit arithmetic, a question classifier and the evaluation code.
The benefit arithmetic is the one place where a wrong answer could cost a student real money, so its tests were written before the code. The rules are short:
That last rule matters most. One test takes a 95.00 dollar charge for a service with a schedule fee of 41.20 and checks that the benefit comes from the fee, not the charge, which leaves a gap of 53.80. Another takes the schedule fee away and checks that the answer is no answer. A calculator that guesses is worse than one that says it does not know.
Our invoices are synthetic. One teammate wrote the script that generates them, and I wrote the code that reads them. The README names a reviewer for every file. Beside the generator it says a different teammate reviews it, and never me.
The reason is simple. Two of the invoice layouts were set aside to score my extractor. If I read the generator, I see what those layouts look like, and the score stops meaning anything. So in a three person repository there is one file I do not open.
The plan gives me the mirror image of that job on the safety side. I write the hidden test questions for the chat’s safety gate, which a teammate builds, and they stay out of the repository until evaluation day. Building against the set you are scored on is not a measurement.
The chat follows the same idea. Every question goes through three separate steps: a review of the question, a draft written only from the member guide, and an audit of the draft. Quotes are checked against the source PDF. If any step fails, the student gets a fixed message, not a best guess.
Emergencies, symptoms and “will my policy pay for this” are stopped before anything is looked up, and pointed to the right service instead. The wording is always “this document shows”, never “you are covered”, because only the insurer can say that.
And no real document goes in, ever. No real invoice, policy number or student. That is a privacy rule, and it is also why the build was small enough to finish.
It is a prototype. Tests that pass with a stand-in for the model do not prove the live model is accurate, and our README says so. Parts of the planned study, including the labels coded by two people, are not finished and are not presented as finished.
What I took from the project is an order of work: decide what the system must refuse to do, and what each person must not see, before deciding what it should do.
The code is on GitHub, and there is a short summary on the work page.
]]>The short answer is a lot. In crashes where exactly one driver was killed or seriously injured, it was the driver of the lighter car two times out of three. After adjusting for driver age and sex, vehicle age, seatbelt use and point of impact, each 10% of extra weight went with about 20% lower odds for that car’s driver.
The full case study has the charts and the numbers. This note is about how we got there.
People who drive small cars are not the same people who drive large utes, and they do not drive on the same roads at the same times. A model that compares all drivers has to adjust for every one of those differences.
We avoided most of that by comparing the two drivers inside the same crash. They share the road, the weather, the speed limit and the moment, so none of those can explain why one was hurt and the other was not. The method is conditional logistic regression.
It has a cost. A crash only tells you something when the two outcomes differ. We started with 57,217 crashes, and 12,078 of them did the work.
The shape. Our plan said the risk would change by a fixed amount per 100 kg. I tested that straight line against a flexible curve before trusting it, and it did not hold. What matters is the ratio of the two weights. An extra 340 kg roughly doubles the odds between two small cars and adds about half between two large ones.
The confounder. We expected vehicle age to explain part of the weight effect, because older cars are lighter and less safe. It explained almost nothing. Driver sex explained two thirds of the shift. Women in the data drove lighter cars and had higher odds of serious injury, so weight had been taking the credit for some of that.
The codes. A speed zone of 999 is not a very fast road. It means unknown. We read every column as text and checked each one against the data dictionary before turning anything into a number.
This is an association in police-reported data. Heavy vehicles are also taller and stiffer, and without make and model we cannot separate weight from design.
It is also only the driver’s side of the story. Other research has found that heavier vehicles are deadlier for the pedestrians they hit. A heavier fleet protects the people inside it and costs the people outside it, and any honest summary needs both halves.
]]>