i find it sad chatgpt & cousins weights are static and context is limited (not just in # of tokens but in how deeply the context can influence the next token & tendency to collapse) ... vs a relationship w a human over time builds so so so much context, good & bad
idk if finetunes are the answer or not- if they are we need a better ux for choosing which interactions to train on & periodically auto-finetune again, while owning & porting our datasets between models as they become obsolete...
& ime models aren't innately that good at taking concise relationally-relevant notes so they can be re-inserted to context later... like, you want the texture, the vibe of a person as well as the right biographical details & recent history to inform your next tokens w them