Spark supports multiple encryption schemes. You can use client side encryption, server side encryption, etc. What wasn’t working for me for a long time is reading encrypted data and writing as a plain text. Before reading I was configuring encryption and this was…
Today I’m going to share my configuration for running custom Anaconda Python with DGL (Deep Graph Library) and mxnet library, with GPU support via CUDA, running in Spark hosted in EMR. Actually, I have Redshift configuration as well, with support for gensim, tensorflow,…
Last time we saw how to connect to Redshift from Spark running in EMR. Provided solution was nice but allowed for reading data only. Sometimes we might want to run any DDL or DML query, not only simple read statements. To do that,…
Today I’ll share my configuration for Spark running in EMR to connect to Redshift cluster. First, I assume the cluster is accessible (so configure virtual subnet, allowed IPs and all network stuff before running this). I’m using Zeppelin so I’ll show two interpreters…
Recently I was investigating machine waking up from sleep without any user interaction. It was a bit irritating because user was putting his PC to sleep and leaving it for a day just to notice in the evening that the machine woke up…
This is the fifteenth part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C# Windows supports multiple desktops for a long time. We…
I was working with seemingly simple load job in Redshift. I was generating some data and wanted to store the output in separate table, just to persist it for debugging purposes. However, since the job could be reran if needed, I wanted to…
Recently I was working with the following code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.stream.Collectors; public class Test { public static void main(String[] args) { List<String> strings = new ArrayList<>(); List<Item> items = strings.stream().map(item -> new Item(new HashMap())).collect(Collectors.toList()); } } class Item { public Map<String, String> metadata; Item(Map<String, String> metadata) { this.metadata = metadata; } } |
I was compiling it with
|
1 2 3 |
java version "1.8.0_221" Java(TM) SE Runtime Environment (build 1.8.0_221-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode) |
on Windows 10 x64. It wasn’t working because of the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
/tmp/java_fR1LWz/Test.java:10: warning: [unchecked] unchecked method invocation: constructor <init> in class Item is applied to given types List<Item> items = strings.stream().map(item -> new Item(new HashMap())).collect(Collectors.toList()); ^ required: Map<String,String> found: HashMap /tmp/java_fR1LWz/Test.java:10: warning: [unchecked] unchecked conversion List<Item> items = strings.stream().map(item -> new Item(new HashMap())).collect(Collectors.toList()); ^ required: Map<String,String> found: HashMap /tmp/java_fR1LWz/Test.java:10: warning: [unchecked] unchecked method invocation: method map in interface Stream is applied to given types List<Item> items = strings.stream().map(item -> new Item(new HashMap())).collect(Collectors.toList()); ^ required: Function<? super T,? extends R> found: Function<String,Item> where T,R are type-variables: T extends Object declared in interface Stream R extends Object declared in method <R>map(Function<? super T,? extends R>) /tmp/java_fR1LWz/Test.java:10: warning: [unchecked] unchecked call to <R,A>collect(Collector<? super T,A,R>) as a member of the raw type Stream List<Item> items = strings.stream().map(item -> new Item(new HashMap())).collect(Collectors.toList()); ^ where R,A,T are type-variables: R extends Object declared in method <R,A>collect(Collector<? super T,A,R>) A extends Object declared in method <R,A>collect(Collector<? super T,A,R>) T extends Object declared in interface Stream /tmp/java_fR1LWz/Test.java:10: error: incompatible types: Object cannot be converted to List<Item> List<Item> items = strings.stream().map(item -> new Item(new HashMap())).collect(Collectors.toList()); ^ 1 error 4 warnings |
You can try reproducing the issue at compilejava.net, it throws the error at the moment. I…
This is a short story of fixing Tomcat in an EMR cluster. There is a CVE about Tomcat 8 hole. I’m using EMR 5.12.2. Let’s try installing CVE patch as a bootstrap action with
|
1 2 3 |
sudo yum update -y --releasever=latest --cve CVE-2019-0232 sudo shutdown -r now |
It should fix the mentioned hole but after…
This is the fourth part of the JVM Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 — Getting object address Typical locking pattern in Java (and other languages, even outside them JVM…
This is the third part of the JVM Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 — Getting object address Erasure in Java seems pretty easy but sometimes it has unexpected consequences.…
Today we will remove Content Security Policy (CSP) header from pages hosted in web view. Technically it doesn’t matter which web view you use (whether it is IE, dotnetbrowser, CEF or whatever else). It all goes down to the same: we need to…
This is the fourteenth part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C# We know that when we call virtual method we…
This is the second part of the JVM Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 — Getting object address Last time we saw how to read object address. We can use…
This is the first part of the JVM Inside Out series. For your convenience you can find other parts using the links below: Part 1 — Getting object address Part 2 — Reading object content Part 3 — Java raw type trickery Part…
This is the fifth part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Today I include some various notes around remote connectivity…
This is the fourth part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Sometimes when I RDP into other machine and forget…
This is the third part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Today we will enable PowerShell Remoting in Windows. This…
This is the second part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows Last week we saw how to configure automatic SSH…
This is the first part of the Availability Anywhere series. For your convenience you can find other parts using the links below: Part 1 — Connecting to SSH tunnel automatically in Windows Part 2 — Connecting to VPN automatically in Windows Part 3…
