The moment Anthony cries out that he has sacrificed “everything” and that she has “no idea what it has cost him,” you see Violet literally falter. She almost recoils. The harshness of his voice makes her realize that this is not just a tantrum, but a deep, years-long buildup of repressed rage.
You can see in her eyes that she knows exactly what he is referring to. As you previously noted: she was the one in Season 1 who reminded him of his “duty” over his “lusts” for Siena. At that moment in the dining room, the mask of the “concerned mother” slips, and you see the realization that she is the architect of his emotional prison. She sees the man she “broke” in order to save the Viscount.
What makes this scene so tragic is that she cannot touch or comfort him at that moment. The wall between them – which she helped build – has become too high. She looks at her son with a mixture of guilt and sorrow, but Anthony resolutely turns and walks away, leaving her alone in the empty, cold dining room.
In Season 1, she could still ignore his grief over Siena or dismiss it as a “phase,” but in this scene, that is no longer possible. The raw pain in his voice forces her to face the damage. This is actually the starting point of her process that eventually leads to those (as you described them, half-hearted) apologies in the final episode.