IT, operating systems, maths, and more.
This is the seventeenth part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C# Last time we saw how to abuse type system. Today we are going to do this to serialize non-serializable type (kind … Continue reading .NET Inside Out Part 17 — Abusing types to serialize non-serializable type
This is the sixteenth part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C# Today we are going to play with type system to see if we can break it. Let’s take this code: We … Continue reading .NET Inside Out Part 16 — Abusing type system
Spark supports multiple encryption schemes. You can use client side encryption, server side encryption, etc. What wasn’t working for me for a long time is reading encrypted data and writing as a plain text. Before reading I was configuring encryption and this was working fine. However, writing as plain text didn’t work (data was encrypted), … Continue reading Data encryption in s3 in Spark in EMR with multiple encryption schemes
Today I’m going to share my configuration for running custom Anaconda Python with DGL (Deep Graph Library) and mxnet library, with GPU support via CUDA, running in Spark hosted in EMR. Actually, I have Redshift configuration as well, with support for gensim, tensorflow, keras, theano, pygpu, and cloudpickle. You can also install more libraries if … Continue reading Running Anaconda with DGL and mxnet on CUDA GPU in Spark running in EMR
Last time we saw how to connect to Redshift from Spark running in EMR. Provided solution was nice but allowed for reading data only. Sometimes we might want to run any DDL or DML query, not only simple read statements. To do that, we need to connect to Redshift directly over JDBC. I assume you … Continue reading Running any query in Redshift or JDBC from Spark in EMR