Running Anaconda with DGL and mxnet on CUDA GPU in Spark running in EMR

March 28, 2020November 8, 2019 ~ afish ~ Leave a comment

Today I’m going to share my configuration for running custom Anaconda Python with DGL (Deep Graph Library) and mxnet library, with GPU support via CUDA, running in Spark hosted in EMR. Actually, I have Redshift configuration as well, with support for gensim, tensorflow, keras, theano, pygpu, and cloudpickle. You can also install more libraries if … Continue reading Running Anaconda with DGL and mxnet on CUDA GPU in Spark running in EMR

Running any query in Redshift or JDBC from Spark in EMR

March 21, 2020March 19, 2020 ~ afish ~ Leave a comment

Last time we saw how to connect to Redshift from Spark running in EMR. Provided solution was nice but allowed for reading data only. Sometimes we might want to run any DDL or DML query, not only simple read statements. To do that, we need to connect to Redshift directly over JDBC. I assume you … Continue reading Running any query in Redshift or JDBC from Spark in EMR

Connecting to Redshift from Spark running in EMR

March 14, 2020November 8, 2019 ~ afish ~ Leave a comment

Today I’ll share my configuration for Spark running in EMR to connect to Redshift cluster. First, I assume the cluster is accessible (so configure virtual subnet, allowed IPs and all network stuff before running this). I’m using Zeppelin so I’ll show two interpreters configured for the connection, but the same thing should work with standalone … Continue reading Connecting to Redshift from Spark running in EMR

Erasure type inference issue in Java

February 15, 2020November 15, 2019 ~ afish ~ Leave a comment

Recently I was working with the following code: I was compiling it with on Windows 10 x64. It wasn’t working because of the following: You can try reproducing the issue at compilejava.net, it throws the error at the moment. I was almost sure that it was a bug in javac, especially that the code was … Continue reading Erasure type inference issue in Java

JVM Inside Out Part 4 — Locks and out of band exceptions

February 1, 2020September 20, 2019 ~ afish ~ Leave a comment

This is the fourth part of the JVM Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 — Getting object address Typical locking pattern in Java (and other languages, even outside them JVM ecosystem) looks like this: Simple enough, nothing should break here. However, there … Continue reading JVM Inside Out Part 4 — Locks and out of band exceptions