Data encryption in s3 in Spark in EMR with multiple encryption schemes

Spark supports multiple encryption schemes. You can use client side encryption, server side encryption, etc. What wasn’t working for me for a long time is reading encrypted data and writing as a plain text. Before reading I was configuring encryption and this was…

Running Anaconda with DGL and mxnet on CUDA GPU in Spark running in EMR

Today I’m going to share my configuration for running custom Anaconda Python with DGL (Deep Graph Library) and mxnet library, with GPU support via CUDA, running in Spark hosted in EMR. Actually, I have Redshift configuration as well, with support for gensim, tensorflow,…

Running any query in Redshift or JDBC from Spark in EMR

Last time we saw how to connect to Redshift from Spark running in EMR. Provided solution was nice but allowed for reading data only. Sometimes we might want to run any DDL or DML query, not only simple read statements. To do that,…

Connecting to Redshift from Spark running in EMR

Today I’ll share my configuration for Spark running in EMR to connect to Redshift cluster. First, I assume the cluster is accessible (so configure virtual subnet, allowed IPs and all network stuff before running this). I’m using Zeppelin so I’ll show two interpreters…

UpdateOrchestrator waking up the machine

Recently I was investigating machine waking up from sleep without any user interaction. It was a bit irritating because user was putting his PC to sleep and leaving it for a day just to notice in the evening that the machine woke up…

.NET Inside Out Part 15 — Starting process on different desktop

This is the fifteenth part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C# Windows supports multiple desktops for a long time. We…

Aborted transactions in Redshift due to Serializable isolation level

I was working with seemingly simple load job in Redshift. I was generating some data and wanted to store the output in separate table, just to persist it for debugging purposes. However, since the job could be reran if needed, I wanted to…

Erasure type inference issue in Java

Recently I was working with the following code:

I was compiling it with

on Windows 10 x64. It wasn’t working because of the following:

You can try reproducing the issue at compilejava.net, it throws the error at the moment. I…

Updating Tomcat in an EMR cluster

This is a short story of fixing Tomcat in an EMR cluster. There is a CVE about Tomcat 8 hole. I’m using EMR 5.12.2. Let’s try installing CVE patch as a bootstrap action with

It should fix the mentioned hole but after…

JVM Inside Out Part 4 — Locks and out of band exceptions

This is the fourth part of the JVM Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 — Getting object address Typical locking pattern in Java (and other languages, even outside them JVM…

JVM Inside Out Part 3 — Java raw type trickery

This is the third part of the JVM Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 — Getting object address Erasure in Java seems pretty easy but sometimes it has unexpected consequences.…

Removing Content Security Policy header from pages hosted in web view

Today we will remove Content Security Policy (CSP) header from pages hosted in web view. Technically it doesn’t matter which web view you use (whether it is IE, dotnetbrowser, CEF or whatever else). It all goes down to the same: we need to…

.NET Inside Out Part 14 — Calling virtual method without dynamic dispatch

This is the fourteenth part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C# We know that when we call virtual method we…

JVM Inside Out Part 2 — Reading object content

This is the second part of the JVM Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 — Getting object address Last time we saw how to read object address. We can use…

JVM Inside Out Part 1 — Getting object address

This is the first part of the JVM Inside Out series. For your convenience you can find other parts using the links below: Part 1 — Getting object address Part 2 — Reading object content Part 3 — Java raw type trickery Part…

Availability Anywhere Part 5 — Various remote connection things

This is the fifth part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Today I include some various notes around remote connectivity…

Availability Anywhere Part 4 — Fixing LogonUi hang

This is the fourth part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Sometimes when I RDP into other machine and forget…

Availability Anywhere Part 3 — How to enable PowerShell remoting in Windows

This is the third part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Today we will enable PowerShell Remoting in Windows. This…

Availability Anywhere Part 2 — Connecting to VPN automatically in Windows

This is the second part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Last week we saw how to configure automatic SSH…

Availability Anywhere Part 1 — Connecting to SSH tunnel automatically in Windows

This is the first part of the Availability Anywhere series. For your convenience you can find other parts using the links below: Part 1 — Connecting to SSH tunnel automatically in Windows Part 2 — Connecting to VPN automatically in Windows Part 3…