I've been wondering how Vengeance worked for a while. Since you don't actually get a buff it can be pretty hard to tell exactly what is going on. Courtesy of the rawr tank modeling discussion on Elistist Jerks, I have my answer, and it is stranger than I could have guessed.
Your maximum vengeance value is, as a blue post indicated not too long ago, your stamina plus 10% of your base health. It was 10% of your health before they increased the health per stamina to 14, but they didn't want to increase Vengeance when they did that. For reference, a tank in 359s might have a maximum Vengeance value around 13k.1
When you take damage, if you do not have Vengeance, you start a new stack of Vengeance which is 5% of the damage you took. Then, every 2 seconds Vengeance ticks and does one of two things:
If you took damage in those two seconds then your new Vengeance value is 95% of your previous value plus 5% of the damage you took.
If you did not take damage then your new value is your old value minus 10% of the largest that stack of Vengeance has ever been.
So if you are taking continuous damage over time at a given rate, your Vengeance stack will tend towards the amount of damage you are taking per two seconds. That is, if you are taking 10k damage per second then your Vengeance will tend towards 20k. Of course that will be way above your cap, so you will actually trend towards your cap quite strongly.
Of course tanks don't take damage continuously. If a boss is swinging once every 2 seconds, and hitting for your Vengeance cap then that will be good enough. The problem is that you will avoid some of those attacks, causing a 10% decay in Vengeance. That will leave you with a 11,700 Vengeance stack. If you get hit again before the next tick then your Vengeance will be 11,115 plus 5% of the hit amount, so you'll need a hit of at least 37,700 to top you back off on that tick, or, more generally, a hit equal to 2.9 times your maximum Vengeance stack. Two misses will mean you will almost certainly not be back up in one strike.
This means that in practice your actual Vengeance will increase non-linearly with Stamina. The more Stamina you have, the more your maximum stack is going to be and the harder a boss will have to hit you to recover the stack after a miss. Of course higher stamina will mean higher stacks, but more stamina means less value for each point of stamina. Against Maloriak, for example, the marginal AP value of Stamina decays from .88 to .71 as you move from 11k to 13k max Vengeance.2 If you could reach 17k Stamina then it would go into freefall, going from .5 to .3 to .02 value per stamina as you got from 18k to 21k. Then at 22k max Vengeance something strange happens: your marginal return is negative. At some point, more stamina means less average AP from Vengeance.
Why does this happen? Given the long period I ran my simulation over, at some point you will dodge infrequently enough that your Vengeance stack will approach Maloriak's average 2 second damage. This will be the high value for your Vengeance stack, and will define the amount of Vengeance you lose if you don't take damage during a 2 second window. The higher this is, the more AP you lose when you successfully dodge an attack. If you are not consistently hovering near your maximum Vengeance stack then you are not getting much benefit from having a higher maximum, but you are still going to end up suffering the drawback of the higher stack size.
If you had no maximum Vengeance against Maloriak hitting you all day, you would end up with around 14.4k Vengeance - at whatever level of armor my Maloriak tank had last time we beat him. But if your Maximum Vengeance stack is around 21k, you will get an average of 15.7k Vengeance. You start doing better than no maximum at around 16k maximum, which is probably attainable this expansion, though obviously not while Maloriak is still relevant.
I don't think there are any bosses currently where more stamina would mean less Vengeance at intended gear levels, but I think Ignacious would close if it weren't for the fact that he is only one phase of the fight and leaves quickly. He has all the right ingredients: short periods of intense damage followed by drawn out periods of mild damage with occasional times where he stops attacking to guarantee a large decay of Vengeance. I can certainly construct reasonable sounding bosses in my head where appropriate levels of Stamina would pass the point where more Stamina means less Vengeance.3
Is this terribly important? I'm not sure. Apparently modelling tools are advising bear druids to gem for stamina, and I know returns from Vengeance are a part of that. I'm sure that advanced modelling tools are using reasonable estimates of Vengeance return for stamina points, but estimates can be reasonable at best, they cannot be accurate in a broad since because there is too much variation based on fight mechanics. I wonder if this is really the best advice a bear can get.
This would be an easy problem to fix. The value of the 10% decays for taking no damage should be set at the last time you took Vengeance damage, not the maximum size your stack has reached. Alternatively, there could simply be a 5% of current value decay every 2 seconds as there is when you do get hit right now, and have it rapidly decay and fall off when you are out of combat. If they don't want it to be that strong then they could simply lower the maximum value.
The current system is just a little too bizarre. If it gave great results then I could accept some strange math, but it gives wacky results. Wacky processes that give wacky results should be replaced with simple processes that give acceptable results.
1. This is the cap for a tank who is gemming heavily for Stamina. Depending on tank class, this might not be a good idea, so that cap might be much lower.
2. Using Maloriak's physical attacks hitting you for a day. In reality he has breaks in his attacks and you'll have to rebuild your stack, so it will be lower than this, spiking temporarily higher during red phase.
3. Imagine a twins fight. We'll make them faceless because I like faceless. So one twin is the caster twin and the other is the beater twin. You put on tank on each. The caster stands and nukes his tank with shadow bolts every 2.6 seconds (2 seconds but he is susceptible to cast slows), but sometimes he does cone of death centred on a random raid member with a 3 second cast (3.9 seconds for real). Every 45-60 seconds one of the twins puts up a shield and channels a spell on the other that makes him do way more damage, increasing over time. You break through the shield and interrupt before the tank on that twin dies. There are adds, void zones, explosions or whatever to do raid damage and keep people reacting to things.
In this fight, when the caster twin is being powered up he would do lots of damage and stack his tank to maximum Vengeance. Then his relatively lighter damage that comes more than every 2 seconds (the healers spend a lot of time on the raid) coupled with taking 3.9 seconds off hitting the tank now and then, would allow the stack to suffer from significant decay. The higher the maximum, the more the tank would be losing, but the stack would never fall off to reset the highest value for that stack because the damage is coming in too consistently.
This sounds like an encounter that could exist. It is an unlikely but believable mix of ingredients that allows Stamina to decrease Vengeance stack size.