#banlagHC2014

[Yavool]

I always seem to qualify these posts with "I'm no programmer, but..." or "I'm no developer, but..." or "I'm no computer expert, but..." (which basically shows my general ignorance toward computer science); nonetheless, here I go:

I'm no server expert, but...I personally have noticed that when Herocraft switched servers back towards the end of the world of Bastion, the lag issue seemed to become worse than it was prior. Now, two wipes later and since the release of Aegis/Sanctuary, I have noticed the lag spikes (for me) to have hit epic proportions, where at times (especially during active boosts) the spikes will last 7-12 seconds long, and will be followed by a [<Insert your name here> just fell to the hunger of the horde!]

I used to run a private family server (just my family & close friends) hosted off my ASUS laptop before joining HC and never experienced lag for myself or my friends/family, but beyond that, I do not know how big-time servers work. With that in mind, I wonder...

1) could someone please explain to me how Minecraft servers work on a larger scale, such as Herocraft?

2) what are the major causes of lag?

3) what does a "lag spike" signify?

4) when I play on other large scale MMORPG servers with thousands of players on at a time (World of Warcraft, League of Legends, Dungeons & Dragons Online, the Battlefield franchise, etc.), I rarely notice lag at the level that I do in HC...what is the difference?

5) at the end of the day, what would be the ideal situation that would eliminate lag altogether (or 98%+) on Herocraft?

I inquire about this because I wish to have the best Herocraft experience for myself (and for everyone possible). Thank you for your help!

-yav

 

[Kainzo]

@Coders - can handle this

 

[yeelowsnow]

Step 1. Buy more loot crates.
Step 2. Kainzo hires developers to figure out the lag.
Step 3. ???
Step 4. Profit.


But really, this is what I've been lead to believe and is not 100% true or correct:

1. Players connect to a central server that hosts the map, plugins and player data. This server is run continuously 24/7, sans restarts and unforeseen downtime. In the case of Herocraft, there are 3? servers at the moment. Aegis, Sanctuary and Central are all on separate physical machines(I hope). Theoretically each machine would have its own set of CPU, RAM and Disk Array(HDD). Another possible configuration entails the usage of virtual machines. That means that the 3 maps(machines) could be hosted on one physical machine. They would then be splitting their usage onto one physical machine. That would greatly increase the stress on the server and components.
   1. I believe and hope that Herocraft is running 3 dedicated machines and not a single machine with virtual machines split across it.
2. entities, saving, memory leaks, etc
3. cpu processes are hanging/stalling trying to either complete a calculation or do something else
4. Dedicated servers, with people whose sole job is to maintain those servers. And when I say dedicated servers, these are datacenter size things. You don't server hundreds, let alone millions of players off of a rack in our basement.
   1. More and more games are moving their servers to AWS which are nearly infinitely scale-able(not really infinitely) in order to more easily deal with increased server loads.
5. IMO solid dev team thats job is only to find and fix the code/memleaks/hardware/anything that is causing lag

Kainzo told me a few days ago that the rubberbanding/freezes are due to the I/O Processes bottoming out to 0% randomly. Hopefully the team can find whats causing that.

 

[LordZelkova]

So I'm not the only one rubber banding and unable to break blocks half the time?

 

[yeelowsnow]

So I'm not the only one rubber banding and unable to break blocks half the time?

no, not at all.

 

[CoolBeans279]

no, not at all.

Really? i haven't gotten anything other than the little lag spikes, otherwise its my internet derping.

 

[Dakinara]

#nomobsonpvpfor1dayjustforkickstoseehowiteffectslagimeanwhynot

 

[what777]

This might be my imagination... but me and w0nder feel like that the server laggs everytime we use /clan lookup on a player.

 

[Delfofthebla]

I'll preface by saying:

I may be a programmer, but just because I am does not mean I know everything about Networks, Servers, or anything about how any game handles their "server processing". It also means I don't know why some games would have no lag, and others would. In my time as a programmer, I have never been the guy to set up the logic for these things. I know absolutely nothing about Net code or how it works.

If I had been working with these things, I'm sure I'd be able to relay more accurate information. Alas, I have not, so my knowledge is limited and/or slightly off-point. Many of you might know more than myself. However, I have worked with Minecraft, and I have a general idea of why it's so god damn shit when it comes to lag.

To quote:

Step 1. Buy more loot crates.
Step 2. Kainzo hires developers to figure out the lag.
Step 3. ???
Step 4. Profit.


But really, this is what I've been lead to believe and is not 100% true or correct:

1. Players connect to a central server that hosts the map, plugins and player data. This server is run continuously 24/7, sans restarts and unforeseen downtime. In the case of Herocraft, there are 3? servers at the moment. Aegis, Sanctuary and Central are all on separate physical machines(I hope). Theoretically each machine would have its own set of CPU, RAM and Disk Array(HDD). Another possible configuration entails the usage of virtual machines. That means that the 3 maps(machines) could be hosted on one physical machine. They would then be splitting their usage onto one physical machine. That would greatly increase the stress on the server and components.
   1. I believe and hope that Herocraft is running 3 dedicated machines and not a single machine with virtual machines split across it.
2. entities, saving, memory leaks, etc
3. cpu processes are hanging/stalling trying to either complete a calculation or do something else
4. Dedicated servers, with people whose sole job is to maintain those servers. And when I say dedicated servers, these are datacenter size things. You don't server hundreds, let alone millions of players off of a rack in our basement.
   1. More and more games are moving their servers to AWS which are nearly infinitely scale-able(not really infinitely) in order to more easily deal with increased server loads.
5. IMO solid dev team thats job is only to find and fix the code/memleaks/hardware/anything that is causing lag

Kainzo told me a few days ago that the rubberbanding/freezes are due to the I/O Processes bottoming out to 0% randomly. Hopefully the team can find whats causing that.

Even with my limited knowledge, I can say that I generally agree on all points.

However, there's a bit more to say on the topic. Any time you're on HC and you run into the question: "What's causing all the lag?" I could probably only have the answer: "I have no fucking idea."

But I'll get back to that. First, in case yeelowsnow's explanation did not make it clear:

World of Warcraft has a MASSIVE datacenter. We're talking thousands of servers. Sometime in between 2006-2010 I believe it was stated that World of Warcraft's servers are more powerful than the ones used by fucking NASA.

Compare that to Herocraft, which is dealing with 1 server per world, it should be no surprise that Herocraft cannot perform as effectively as a large scale AAA ranked MMORPG.

Now this is where my ignorance comes in. I may be incorrect in stating this, but I believe that World of Warcraft (and other MMOs) are able to use these "thousands of servers" for different things. Perhaps one server handles all the logic of combat, one server may handle player's locations in the world, and another is tracking all the changes made to all characters (looting / deleting items)

If I am correct in assuming that, you can see the fault of Minecraft/Herocraft right there. The best anyone can do is assign ONE server to ONE world. I do not believe anyone has found a workaround for this yet. That means that all character locations, combat logic, character changes, and not to mention anyone modifying ANY part of the whole fucking world. It's a lot to take in, and I don't think that you will ever reach a point of "zero lag" because of this.​

So, most of HC's lag stems from the Minecraft engine itself. Even if Herocraft has 3 servers, 5 servers, or 100 servers, the most we can ever do is put one server on one world. The only time this is even beneficial is if we split the playerbase across multiple worlds. (This is what network servers like Shotbow do, by the way.)

Minecraft is simply not powerful enough to allow server owners to delegate their processing to other systems. Additionally, Minecraft does not have an elegant form of plugin compatibility. By default, there is absolutely ZERO compatbility between plugins. It's impossible to get multiple plugins to work together or "talk" to each other. Hell, your plugin can't even talk to itself in some cases! And if one plugin changes the same thing another plugin does, bad things happen.

This is why Bukkit exists.

Bukkit is a shared resource that plugins use to achieve compatibility with themselves, as well as other plugins. It also saves plugin developers a lot of time trying to figure out how the flying fuck Minecraft even works.

(WARNING: Incoming ignorant babble based off of loose interpretations and understandings of something much more complex than anything I've ever worked with.)

This compatibility comes at a cost however. A lot of extra processing and nonsense gets churned out when plugins "utilize" Bukkit. Additionally, the more plugins you have, the more garbage you're processing. It adds up pretty quickly... So, inherently, the more plugins you have, the more lag you're causing. The more you use Bukkit, the more processing you'll have to do behind the scenes. The more you use bukkit, the more lag you cause. (And since Bukkit is mostly used to modify/interact with vanilla Minecraft, anything that "strongly" modifies Vanilla MC will be processing a lot of stuff, and possible forcing other plugins to do extra unnecessary work.)​

Herocraft does just that. It modifies pretty much ALL forms of damage that can ever be dealt by anyone or anything. It modifies how experience works, how enchanting works, and now, how crafting works. All of this extra "modifying" that we do is a LOT that the server has to work with. If you ever here Kainzo or anyone else blaming "entities" for lag, this is the primary reason for that. We have to process every god damn entity through our damage handlers, and any other nonsense we do, like modifying their HP / Loot Tables / Granted XP.

Of course, we also have to deal with memory leaks caused by any of our plugins (whether that be Heroes or something like LWC), and that can be a giant pain in the ass.

=====

To Summarize, the lag will always be around. Sometimes it's Minecraft's fault, sometimes it's Bukkit's fault, sometimes it's Kainzo's fault, sometimes it's my fault. Anyone that touches any sort of Minecraft related piece of code has a potential to cause 3 metric fucktons of lag and not a single person in the world will be able to know who did it, or why they did it, until they get bit in the face by it and start poking around.

I don't play HC right now so I can't say I know much about the current lag situation, but I'm sure it's something stupid, and there's not much we could do about it until we found out what exactly was causing it. Once we find out, which could take hours to months, you're left with one of the following options:

1) Fix the bug that caused it, decreasing lag.
--Often the hardest solution, and equally often is impossible for us to do from our end. If it's a plugin we control, it's...easier. But if it's an external plugin, our only choice is to notify the developer of the issue and hope he gets around to it.

2) Upgrade the server
--A bandaid solution that often only leads to an extra 30 minutes before the server needs restarting. Sometimes that 30 minutes is enough to sate our sanity, but it's expensive, so Kainzo is less likely to resort to this.

3) Remove the plugin(s) causing the problem.
--Sometimes a valid solution, or, when Heroes is at fault, twiddle our thumbs until someone on the dev team finds a solution to the bug


So yea, there you go. There's probably more to say, and I'm probably not 100% accurate in my statements, but you can hopefully piece together an "idea" of where the lag comes from now.

 

[devotedworker]

I always seem to qualify these posts with "I'm no programmer, but..." or "I'm no developer, but..." or "I'm no computer expert, but..." (which basically shows my general ignorance toward computer science); nonetheless, here I go:

I'm no server expert, but...I personally have noticed that when Herocraft switched servers back towards the end of the world of Bastion, the lag issue seemed to become worse than it was prior. Now, two wipes later and since the release of Aegis/Sanctuary, I have noticed the lag spikes (for me) to have hit epic proportions, where at times (especially during active boosts) the spikes will last 7-12 seconds long, and will be followed by a [<Insert your name here> just fell to the hunger of the horde!]

I used to run a private family server (just my family & close friends) hosted off my ASUS laptop before joining HC and never experienced lag for myself or my friends/family, but beyond that, I do not know how big-time servers work. With that in mind, I wonder...

1) could someone please explain to me how Minecraft servers work on a larger scale, such as Herocraft?

2) what are the major causes of lag?

3) what does a "lag spike" signify?

4) when I play on other large scale MMORPG servers with thousands of players on at a time (World of Warcraft, League of Legends, Dungeons & Dragons Online, the Battlefield franchise, etc.), I rarely notice lag at the level that I do in HC...what is the difference?

5) at the end of the day, what would be the ideal situation that would eliminate lag altogether (or 98%+) on Herocraft?

I inquire about this because I wish to have the best Herocraft experience for myself (and for everyone possible). Thank you for your help!

-yav

Ok! Lets go through each question Even though some great people have explained it very well, but I'll try to use more basic terms!

1) could someone please explain to me how Minecraft servers work on a larger scale, such as Herocraft?
Herocraft and most large servers operate using what I call a "Server-Net" or using a link to ship ( Pun cause herocraft uses ships XD ) users from server to server. This is to divide the playerload equaly and efficently.
2) what are the major causes of lag?
Players! As I quickly learned from my ( Mildly ) short time of running servers that players and them constantly loading chunks lags the server alot. Surprisingly its not the Anti-Xray or the things you would normaly suspect. ( Though Hero's does cause some lag I assume ) Dont forget bugs and memory leaks!
3) what does a "lag spike" signify?
Actualy its quite simple. What happens is the system quickly pauses everything ( Serverwise ) and tries to proccess the overflow of information and then quickly resumes. It also skips the tick clock ahead... which is why after lag cactus's may grow after a lag spike >.<
4) when I play on other large scale MMORPG servers with thousands of players on at a time (World of Warcraft, League of Legends, Dungeons & Dragons Online, the Battlefield franchise, etc.), I rarely notice lag at the level that I do in HC...what is the difference?
Well for 1 thing, those games don't constantly have to send chunk information portraying to hundreds of thosands of blocks at once. Most of the time they just have to send player position and Mob info. Not to mention they are run on super computers
5) at the end of the day, what would be the ideal situation that would eliminate lag altogether (or 98%+) on Herocraft?
For us to stop yelling lag so the staff members can figure out whats causing it... Hahaha
In reality it would be for the Dev team to eliminate every memory leak and optimise the system overall. Which they are currently ( Or better be ) doing!

 

[Ultanian]

I'll preface by saying:

I may be a programmer, but just because I am does not mean I know everything about Networks, Servers, or anything about how any game handles their "server processing". It also means I don't know why some games would have no lag, and others would. In my time as a programmer, I have never been the guy to set up the logic for these things. I know absolutely nothing about Net code or how it works.

If I had been working with these things, I'm sure I'd be able to relay more accurate information. Alas, I have not, so my knowledge is limited and/or slightly off-point. Many of you might know more than myself. However, I have worked with Minecraft, and I have a general idea of why it's so god damn shit when it comes to lag.

To quote:

Even with my limited knowledge, I can say that I generally agree on all points.

However, there's a bit more to say on the topic. Any time you're on HC and you run into the question: "What's causing all the lag?" I could probably only have the answer: "I have no fucking idea."

But I'll get back to that. First, in case yeelowsnow's explanation did not make it clear:

World of Warcraft has a MASSIVE datacenter. We're talking thousands of servers. Sometime in between 2006-2010 I believe it was stated that World of Warcraft's servers are more powerful than the ones used by fucking NASA.

Compare that to Herocraft, which is dealing with 1 server per world, it should be no surprise that Herocraft cannot perform as effectively as a large scale AAA ranked MMORPG.

Now this is where my ignorance comes in. I may be incorrect in stating this, but I believe that World of Warcraft (and other MMOs) are able to use these "thousands of servers" for different things. Perhaps one server handles all the logic of combat, one server may handle player's locations in the world, and another is tracking all the changes made to all characters (looting / deleting items)

If I am correct in assuming that, you can see the fault of Minecraft/Herocraft right there. The best anyone can do is assign ONE server to ONE world. I do not believe anyone has found a workaround for this yet. That means that all character locations, combat logic, character changes, and not to mention anyone modifying ANY part of the whole fucking world. It's a lot to take in, and I don't think that you will ever reach a point of "zero lag" because of this.​

So, most of HC's lag stems from the Minecraft engine itself. Even if Herocraft has 3 servers, 5 servers, or 100 servers, the most we can ever do is put one server on one world. The only time this is even beneficial is if we split the playerbase across multiple worlds. (This is what network servers like Shotbow do, by the way.)

Minecraft is simply not powerful enough to allow server owners to delegate their processing to other systems. Additionally, Minecraft does not have an elegant form of plugin compatibility. By default, there is absolutely ZERO compatbility between plugins. It's impossible to get multiple plugins to work together or "talk" to each other. Hell, your plugin can't even talk to itself in some cases! And if one plugin changes the same thing another plugin does, bad things happen.

This is why Bukkit exists.

Bukkit is a shared resource that plugins use to achieve compatibility with themselves, as well as other plugins. It also saves plugin developers a lot of time trying to figure out how the flying fuck Minecraft even works.

(WARNING: Incoming ignorant babble based off of loose interpretations and understandings of something much more complex than anything I've ever worked with.)

This compatibility comes at a cost however. A lot of extra processing and nonsense gets churned out when plugins "utilize" Bukkit. Additionally, the more plugins you have, the more garbage you're processing. It adds up pretty quickly... So, inherently, the more plugins you have, the more lag you're causing. The more you use Bukkit, the more processing you'll have to do behind the scenes. The more you use bukkit, the more lag you cause. (And since Bukkit is mostly used to modify/interact with vanilla Minecraft, anything that "strongly" modifies Vanilla MC will be processing a lot of stuff, and possible forcing other plugins to do extra unnecessary work.)​

Herocraft does just that. It modifies pretty much ALL forms of damage that can ever be dealt by anyone or anything. It modifies how experience works, how enchanting works, and now, how crafting works. All of this extra "modifying" that we do is a LOT that the server has to work with. If you ever here Kainzo or anyone else blaming "entities" for lag, this is the primary reason for that. We have to process every god damn entity through our damage handlers, and any other nonsense we do, like modifying their HP / Loot Tables / Granted XP.

Of course, we also have to deal with memory leaks caused by any of our plugins (whether that be Heroes or something like LWC), and that can be a giant pain in the ass.

=====

To Summarize, the lag will always be around. Sometimes it's Minecraft's fault, sometimes it's Bukkit's fault, sometimes it's Kainzo's fault, sometimes it's my fault. Anyone that touches any sort of Minecraft related piece of code has a potential to cause 3 metric fucktons of lag and not a single person in the world will be able to know who did it, or why they did it, until they get bit in the face by it and start poking around.

I don't play HC right now so I can't say I know much about the current lag situation, but I'm sure it's something stupid, and there's not much we could do about it until we found out what exactly was causing it. Once we find out, which could take hours to months, you're left with one of the following options:

1) Fix the bug that caused it, decreasing lag.
--Often the hardest solution, and equally often is impossible for us to do from our end. If it's a plugin we control, it's...easier. But if it's an external plugin, our only choice is to notify the developer of the issue and hope he gets around to it.

2) Upgrade the server
--A bandaid solution that often only leads to an extra 30 minutes before the server needs restarting. Sometimes that 30 minutes is enough to sate our sanity, but it's expensive, so Kainzo is less likely to resort to this.

3) Remove the plugin(s) causing the problem.
--Sometimes a valid solution, or, when Heroes is at fault, twiddle our thumbs until someone on the dev team finds a solution to the bug


So yea, there you go. There's probably more to say, and I'm probably not 100% accurate in my statements, but you can hopefully piece together an "idea" of where the lag comes from now.

There is another way to decrease the lag...

(@Kainzo make your own game )

 

[Kainzo]

By dumb luck and sheer gut feeling we found a MASSIVE source of the lag (/clan lookup <offlineplayer> - since we have removed this - we've seen a drop in about 50% spikes.

Why was this hard to catch? 1) Multiple servers had the cmd enabled and usable 2) you had to be viewing the console to see it hold on this cmd 3) it didnt hold the server on ONLINE players. 4) No where did this show up in any of our performance kits, such as JMX / timings - it was missed completely.

This cmd has been restricted until we can make it for ONLINE players only.

 

[yeelowsnow]

This might be my imagination... but me and w0nder feel like that the server laggs everytime we use /clan lookup on a player.

By dumb luck and sheer gut feeling we found a MASSIVE source of the lag (/clan lookup <offlineplayer> - since we have removed this - we've seen a drop in about 50% spikes.

Why was this hard to catch? 1) Multiple servers had the cmd enabled and usable 2) you had to be viewing the console to see it hold on this cmd 3) it didnt hold the server on ONLINE players. 4) No where did this show up in any of our performance kits, such as JMX / timings - it was missed completely.

This cmd has been restricted until we can make it for ONLINE players only.

winner winner @what777

 

[Fjordsen]

lol nvm.

 

[Yavool]

I learned a lot today because of all of your input... thank you for that! I would offer my assistance, but beyond moral support, you might as well ask me to build a rocket to the moon. So, instead ~ keep up the hard work in your efforts to quash the boss mob named Lag!

-yav

 

[Kainzo]

I learned a lot today because of all of your input... thank you for that! I would offer my assistance, but beyond moral support, you might as well ask me to build a rocket to the moon. So, instead ~ keep up the hard work in your efforts to quash the boss mob named Lag!

-yav

Donating / Store purchases help - but not directly into the issue. You should notice a tremendous amount of less lag via the changes recently.

 

[greatman321]

Step 1. Buy more loot crates.
Step 2. Kainzo hires developers to figure out the lag.
Step 3. ???
Step 4. Profit.


But really, this is what I've been lead to believe and is not 100% true or correct:

1. Players connect to a central server that hosts the map, plugins and player data. This server is run continuously 24/7, sans restarts and unforeseen downtime. In the case of Herocraft, there are 3? servers at the moment. Aegis, Sanctuary and Central are all on separate physical machines(I hope). Theoretically each machine would have its own set of CPU, RAM and Disk Array(HDD). Another possible configuration entails the usage of virtual machines. That means that the 3 maps(machines) could be hosted on one physical machine. They would then be splitting their usage onto one physical machine. That would greatly increase the stress on the server and components.
   1. I believe and hope that Herocraft is running 3 dedicated machines and not a single machine with virtual machines split across it.
2. entities, saving, memory leaks, etc
3. cpu processes are hanging/stalling trying to either complete a calculation or do something else
4. Dedicated servers, with people whose sole job is to maintain those servers. And when I say dedicated servers, these are datacenter size things. You don't server hundreds, let alone millions of players off of a rack in our basement.
   1. More and more games are moving their servers to AWS which are nearly infinitely scale-able(not really infinitely) in order to more easily deal with increased server loads.
5. IMO solid dev team thats job is only to find and fix the code/memleaks/hardware/anything that is causing lag

Kainzo told me a few days ago that the rubberbanding/freezes are due to the I/O Processes bottoming out to 0% randomly. Hopefully the team can find whats causing that.

For the questions:

1. We currently use 1 machine that have plenty of power for the whole network (MC servers). The box is just right for HC, not overloaded, not underloaded.

2. In our timing report what use the most 99% of the times are the stupid entities. Blame MC on that unfortunatly.

3. We are maybe gonna offload some parts of the network on another machine to test if it helps.

4.
AWS does cost a fuck-ton of money when you get a powerful machine tho. If we take let's say r3.large which have 15gb ram (The servers are currently configurated to use around 10 each), it cost 126$/month. Multiply that by 3 for HC (for central, pvp & PVE), that gives 504$. Finish it with the bandwidth around 500gb/month (Being a host earlier make me know 150-200+ players servers use around 1TB/month) you add 60$.

So from a normal dedicated where it's around 100-200$/month we get AWS for 3-4x the price heh.

5. As Kainzo said we found one cause of the slowdown with the clan plugin, it helped greatly to reduce the spike. So it's a start

 

[Dakinara]

are mobs the main source of entity lag? could we seriously try turning mobs off the pvp server even if for just a day to see how it works?

I know there are a lot of people that hate the idea for various reasons, but if it had a huge impact on lag, id happily trade mobs out and maybe make a massive pvp experience increase (but only exp for killing rival clans or some such) and also increase conquest exp rewards.

 

[what777]

are mobs the main source of entity lag? could we seriously try turning mobs off the pvp server even if for just a day to see how it works?

I know there are a lot of people that hate the idea for various reasons, but if it had a huge impact on lag, id happily trade mobs out and maybe make a massive pvp experience increase (but only exp for killing rival clans or some such) and also increase conquest exp rewards.

Make PVE the leveling map, and just disable mobs on PVP indeed.

 

[Yavool]

Donating / Store purchases help - but not directly into the issue. You should notice a tremendous amount of less lag via the changes recently.

Sadly, I haven't noticed any difference in the lag spikes (still ranging between 5 and 12 seconds), nor the block lag moments when mining. I wish I did...but empirical data states otherwise!