Data encryption in s3 in Spark in EMR with multiple encryption schemes

Spark supports multiple encryption schemes. You can use client side encryption, server side encryption, etc. What wasn’t working for me for a long time is reading encrypted data and writing as a plain text. Before reading I was configuring encryption and this was working fine. However, writing as plain text didn’t work (data was encrypted), … Continue reading Data encryption in s3 in Spark in EMR with multiple encryption schemes

Running Anaconda with DGL and mxnet on CUDA GPU in Spark running in EMR

Today I’m going to share my configuration for running custom Anaconda Python with DGL (Deep Graph Library) and mxnet library, with GPU support via CUDA, running in Spark hosted in EMR. Actually, I have Redshift configuration as well, with support for gensim, tensorflow, keras, theano, pygpu, and cloudpickle. You can also install more libraries if … Continue reading Running Anaconda with DGL and mxnet on CUDA GPU in Spark running in EMR

Connecting to Redshift from Spark running in EMR

Today I’ll share my configuration for Spark running in EMR to connect to Redshift cluster. First, I assume the cluster is accessible (so configure virtual subnet, allowed IPs and all network stuff before running this). I’m using Zeppelin so I’ll show two interpreters configured for the connection, but the same thing should work with standalone … Continue reading Connecting to Redshift from Spark running in EMR

Aborted transactions in Redshift due to Serializable isolation level

I was working with seemingly simple load job in Redshift. I was generating some data and wanted to store the output in separate table, just to persist it for debugging purposes. However, since the job could be reran if needed, I wanted to remove rows generated previously and add new ones. So I tried the … Continue reading Aborted transactions in Redshift due to Serializable isolation level