"... the generator of human values is not intrinsic to individual humans but is a broader societal phenomenon which is primarily transmitted through linguistic media and hence is easily accessible and learnable by linguistic AI systems like language models, which is also what we observe empirically where LLMs actually have very strong grasps of contemporary morality and moral systems and can apply these in somewhat novel domains. Similarly, we have no particularly strong evidence about how human values would generalize ‘out of distribution’ in a singularity style scenario, and I think there is some fairly strong counter-evidence against the idea that human values have generalized well in the past. for instance our values today are extremely different from those of almost all historical societies and, in many ways, ‘by their lights’ today’s society would be a massive regression in values." worth reading in full https://www.beren.io/2025-11-30-The-Biosingularity-Alignment-Problem-Seems-Harder-than-AI-Alignment/