IT, operating systems, maths, and more.
Recently I was debugging a case of NoSuchMethodError when using the Google Guava library. Conceptually my code was doing the following: In runtime I was using library in version 19 and I was getting the following error: I decompiled the code and got something similar to this If you look closely you can see that … Continue reading Source compatibility issue with Google Guava library
It sometimes happens that your EMR job crashes or hangs indefinitely with no meaningful log. You can try to capture memory dump but it is not very useful when your cluster machines have hundreds gigabytes of memory each. Below are “fixes” which worked for me. If it just crashes with lost slave or lost task, … Continue reading Random notes from crashing and hanging EMR Spark job
When you deploy your package to Amazon Elastic Map Reduce (EMR), you can access the AWS SDK provided by the platform. This gets tricky if you compile your code against different version of SDK because then you may get very cryptic bugs in runtime, like class not found or method not existing. You should always … Continue reading Investigating AWS SDK conflicts in EMR
Today we are going to implement trivial ScheduledExecutorService in C#. Introduction We want to have a thread pool of fixed size which we can use to execute three types of tasks: One time tasks Tasks with fixed delay between executions Tasks with fixed rate of executions Let’s go. Bookkeeping We start with the following: We … Continue reading Trivial ScheduledThreadPoolExecutor in C#
Imagine that you need to load JAR file dynamically in Zeppelin working on your EMR cluster. One easy way is to deploy the file to the instance and load it from there, however, what can you do if you have almost no access to the cluster and the filesystem? You can load the JAR from … Continue reading Dynamically loading JAR file in Zeppelin