Comments on: A reminder about remainder operator (%)

By: Adam Malter

Adam Malter — Fri, 10 Jul 2009 21:03:15 +0000

What is this uint you speak of? Some mystical item from a faraway land?

By: ppow

ppow — Fri, 10 Jul 2009 18:08:30 +0000

About the circular buffer: Try to imagine that your system is trying to compare two algorithms (A and B) against each other by running them with different inputs (i1, i2,…iN) and checking whether they will give the same results (For example A is a brut-force slow solution, B is a fast sophisticated algorithm that you want to check). It is very possible that your jobs queue will look like this:
A(i1), B(i1), A(i2), B(i2), A(i3), B(i3), …, A(iN), B(iN)
so there is always alternating A and B tasks. The execution time of every task is ‘random’, but then it is expected that A(i) >> B(i).

Now try to imagine that you have 100 machines in S. If the system would use your algorithm, then all servers with even IDs would get the A tasks and servers with even IDs would get the tasks B. Therefore odd-ID servers would have much more load and still continue to process the tasks while the even-ID servers would be idle….

I just wanted to show you that there can be a task distribution pattern in J and its impact on processing can be really bad. Basically since the task generator is not really random you do have to expect to have some pattern in J and that is why you should distribute them among the machines in a ‘random’ way:)

…and yes, unsigned int would solve the issue. But then again you would have to know about the issue first and if you did you would probably find a more elegant solution:)

By: antimatroid

antimatroid — Fri, 10 Jul 2009 15:37:49 +0000

Why bother to randomly distribute the jobs if the execution times of those jobs are inherently random? Say you have a queue of jobs (J), and a circular buffer of servers (S) which those jobs may execute upon. For each job j in J, put j on S and make S the next server in the circular buffer. If the jobs are random, then this configuration will be just ‘as random’ as the configuration you choose by route of hashcode.

As a side note- you could have avoided this bug by going with a uint rather than int and your ring would have been Z+ rather than just Z.

By: ppow

ppow — Fri, 10 Jul 2009 15:11:06 +0000

Yeah, random with a defined seed would be the best solution – it would give you a guarantee of an even and fair distribution no matter the number of servers or pattern in task queue.

By: Artur

Artur — Fri, 10 Jul 2009 12:11:16 +0000

It would be quite funny if you have written unit tests with all the task names having positive hashcodes by accident…

By: tk

tk — Fri, 10 Jul 2009 12:03:51 +0000

Relying on hashcode is rather tricky — if any component in the computed hashcode relies on the built-in hashcode of an instance variable, you might be basing your hashcode computation on memory address somewhere.

Have you thought about instantiating a PRNG with a defined seed? Just use random.next() * N and you should get a repeatable, uniform distribution.

By: rajs

rajs — Wed, 08 Jul 2009 16:55:22 +0000

Thanks for sharing this! I can imagine the furstation of debugging and the relief of “Aha!” moment when you figured it. I had similar moments when I made a mistake in commenting out a innocent looking println (part of code cleanup to avoid clogging of log files) and created a infinite loop in production environment bringing all app servers down! It took around 6 hours to figure out that a simple comment of the println has caused this. The println was “System.out.println(”the element is :” + e.nextElement())” which was inside a while (e.hasMoreElements()) loop. Since only the println was commented, the while loop became a infite one. We never bothered to test it as we just commented a simple println. Hard way to learn the importance of testing!

Rajs
http://thoughtsinprogress.co.in