My configuration uses EMR 5.17.2 and CUDA 9.2. When I’m writing it, there is EMR 5.27 available but it comes with the same CUDA version so I presume it should work as well. I’m also using Python 3.7.
First, create a cluster. Do not select mxnet
as a provided library in EMR, we will install it later. As a master node use p3.8xlarge instance type — this instance must have GPU and this is where we will run DGL and mxnet. For slaves you can use anything, I’m going with 19 r3.4xlarge nodes (they don’t have GPU).
We need to install some custom libraries. I am using bootstrap script for that but you can just SSH into the host manually and run this code:
sudo mkdir /mnt/usr-moved sudo mv /usr/local /mnt/usr-moved/ sudo ln -s /mnt/usr-moved/local /usr/ sudo mv /usr/share /mnt/usr-moved/ sudo ln -s /mnt/usr-moved/share /usr/ sudo mkdir /mnt/home sudo chown hadoop.hadoop /mnt/home wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh -O ~/anaconda.sh bash ~/anaconda.sh -b -p /mnt/home/hadoop/anaconda echo -e '\nexport PATH=/mnt/home/hadoop/anaconda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc echo -e '\nexport PYSPARK_PYTHON=/mnt/home/hadoop/anaconda/bin/python' >> $HOME/.bashrc && source $HOME/.bashrc /mnt/home/hadoop/anaconda/bin/conda install -y gensim /mnt/home/hadoop/anaconda/bin/conda install -y tensorflow /mnt/home/hadoop/anaconda/bin/conda install -y keras /mnt/home/hadoop/anaconda/bin/conda install -y theano /mnt/home/hadoop/anaconda/bin/conda install -y pygpu /mnt/home/hadoop/anaconda/bin/conda upgrade -y cloudpickle yes | sudo /mnt/home/hadoop/anaconda/bin/pip install --pre mxnet-cu92mkl /mnt/home/hadoop/anaconda/bin/conda install -y -c dglteam dgl-cuda9.2 /mnt/home/hadoop/anaconda/bin/conda install -y s3fs
First, I’m making a symlink to not fill the disk while installing packages. Then in line 10 I download Anaconda. Finally, lines 15-23 install some additional libraries. Notice that in line 21 I install mxnet compiled for CUDA 9.2, and in line 22 the same for DGL. Also, s3fs is required for nice reading from s3.
When this is done and cluster is created, I replace Python for Zeppelin interpreter to point to /mnt/home/hadoop/anaconda/bin/python
and add Redshift configuration. I do this with the following command line (this you need to run manually after the cluster is created):
sudo /usr/lib/zeppelin/bin/install-interpreter.sh --name jdbc cd /usr/lib/zeppelin/interpreter/jdbc/ sudo wget https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.20.1043/RedshiftJDBC42-no-awssdk-1.2.20.1043.jar
sudo cat | sudo tee /etc/zeppelin/conf/interpreter.json <<'endmsg' { "interpreterSettings": { "2ANGGHHMQ": { "id": "2ANGGHHMQ", "name": "spark", "group": "spark", "properties": { "zeppelin.spark.printREPLOutput": "true", "spark.yarn.jar": "", "master": "yarn-client", "zeppelin.spark.maxResult": "1000", "zeppelin.dep.localrepo": "/usr/lib/zeppelin/local-repo", "spark.app.name": "Zeppelin", "zeppelin.spark.importImplicit": "true", "zeppelin.spark.useHiveContext": "true", "args": "", "spark.home": "/usr/lib/spark", "zeppelin.spark.concurrentSQL": "false", "zeppelin.pyspark.python": "/mnt/home/hadoop/anaconda/bin/python" }, "status": "READY", "interpreterGroup": [ { "name": "spark", "class": "org.apache.zeppelin.spark.SparkInterpreter", "defaultInterpreter": false, "editor": { "language": "scala", "editOnDblClick": false } }, { "name": "pyspark", "class": "org.apache.zeppelin.spark.PySparkInterpreter", "defaultInterpreter": false, "editor": { "language": "python", "editOnDblClick": false } }, { "name": "sql", "class": "org.apache.zeppelin.spark.SparkSqlInterpreter", "defaultInterpreter": false, "editor": { "language": "sql", "editOnDblClick": false } } ], "dependencies": [ { "groupArtifactVersion": "/usr/lib/zeppelin/interpreter/jdbc/RedshiftJDBC42-no-awssdk-1.2.20.1043.jar", "local": false } ], "option": { "remote": true, "port": -1, "perNote": "shared", "perUser": "shared", "isExistingProcess": false, "setPermission": false, "users": [], "isUserImpersonate": false } }, "2AM1YV5CU": { "id": "2AM1YV5CU", "name": "angular", "group": "angular", "properties": {}, "status": "READY", "interpreterGroup": [ { "name": "angular", "class": "org.apache.zeppelin.angular.AngularInterpreter", "defaultInterpreter": false, "editor": { "editOnDblClick": true } } ], "dependencies": [], "option": { "remote": true, "port": -1, "perNote": "shared", "perUser": "shared", "isExistingProcess": false, "setPermission": false, "isUserImpersonate": false } }, "2BRWU4WXC": { "id": "2BRWU4WXC", "name": "python", "group": "python", "properties": { "zeppelin.python": "/mnt/home/hadoop/anaconda/bin/python", "zeppelin.python.maxResult": "1000" }, "status": "READY", "interpreterGroup": [ { "name": "python", "class": "org.apache.zeppelin.python.PythonInterpreter", "defaultInterpreter": false, "editor": { "language": "python", "editOnDblClick": false } } ], "dependencies": [], "option": { "remote": true, "port": -1, "perNote": "shared", "perUser": "shared", "isExistingProcess": false, "setPermission": false, "users": [], "isUserImpersonate": false } }, "2AJXGMUUJ": { "id": "2AJXGMUUJ", "name": "md", "group": "md", "properties": {}, "status": "READY", "interpreterGroup": [ { "name": "md", "class": "org.apache.zeppelin.markdown.Markdown", "defaultInterpreter": false, "editor": { "language": "markdown", "editOnDblClick": true } } ], "dependencies": [], "option": { "remote": true, "port": -1, "perNote": "shared", "perUser": "shared", "isExistingProcess": false, "setPermission": false, "isUserImpersonate": false } }, "2AKK3QQXU": { "id": "2AKK3QQXU", "name": "sh", "group": "sh", "properties": { "shell.command.timeout.millisecs": "60000" }, "status": "READY", "interpreterGroup": [ { "name": "sh", "class": "org.apache.zeppelin.shell.ShellInterpreter", "defaultInterpreter": false, "editor": { "language": "sh", "editOnDblClick": false } } ], "dependencies": [], "option": { "remote": true, "port": -1, "perNote": "shared", "perUser": "shared", "isExistingProcess": false, "setPermission": false, "isUserImpersonate": false } } }, "interpreterBindings": { "2EMW16J14": [ "2ANGGHHMQ", "2AJXGMUUJ", "2AM1YV5CU", "2AKK3QQXU", "2BRWU4WXC" ], "2A94M5J1Z": [ "2ANGGHHMQ", "2AJXGMUUJ", "2AM1YV5CU", "2AKK3QQXU", "2BRWU4WXC" ] }, "interpreterRepositories": [ { "id": "central", "type": "default", "url": "http://repo1.maven.org/maven2/", "releasePolicy": { "enabled": true, "updatePolicy": "daily", "checksumPolicy": "warn" }, "snapshotPolicy": { "enabled": true, "updatePolicy": "daily", "checksumPolicy": "warn" }, "mirroredRepositories": [], "repositoryManager": false }, { "id": "local", "type": "default", "url": "file:///var/lib/zeppelin/.m2/repository", "releasePolicy": { "enabled": true, "updatePolicy": "daily", "checksumPolicy": "warn" }, "snapshotPolicy": { "enabled": true, "updatePolicy": "daily", "checksumPolicy": "warn" }, "mirroredRepositories": [], "repositoryManager": false } ] } endmsg
Now, I need to tune default spark submit options:
sudo cat | sudo tee /etc/zeppelin/conf.dist/zeppelin-env.sh <<'endmsg' export ZEPPELIN_PORT=8890 export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf export ZEPPELIN_LOG_DIR=/var/log/zeppelin export ZEPPELIN_PID_DIR=/var/run/zeppelin export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook export MASTER=yarn-client export SPARK_HOME=/usr/lib/spark export HADOOP_CONF_DIR=/etc/hadoop/conf export LD_LIBRARY_PATH=/usr/local/cuda/lib64/ export CLASSPATH=":/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar" export SPARK_SUBMIT_OPTIONS="--jars=YOUR_JARS_HERE --conf spark.executor.extraClassPath=/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/* --conf spark.driver.extraClassPath=/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/* --conf 'spark.executorEnv.PYTHONPATH=/usr/lib/spark/python/lib/py4j-src.zip:/usr/lib/spark/python/:<CPS>{{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-src.zip' --conf spark.yarn.isPython=true --conf spark.driver.maxResultSize=80G --num-executors 56 --executor-cores 5 --executor-memory 38G --driver-memory 90G --conf 'spark.dynamicAllocation.enabled=false' --conf 'spark.hadoop.fs.s3.canned.acl=BucketOwnerFullControl' --conf 'spark.hadoop.fs.s3.acl.default=BucketOwnerFullControl' --conf 'spark.hadoop.fs.s3.acl=bucket-owner-full-control' --conf 'spark.hadoop.fs.s3n.acl=bucket-owner-full-control' --conf 'spark.hadoop.fs.s3.cse.enabled=false'" export ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo endmsg
This is not the full content! I omit some of my internal settings so generally don’t copy it blindly, just extend the zeppelin-env.sh
file as needed. Important things are:
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/
— this points to CUDA libraries
--conf spark.driver.maxResultSize=80G --num-executors 56 --executor-cores 5 --executor-memory 38G --driver-memory 90G --conf 'spark.dynamicAllocation.enabled=false'
— this configures executors and memory. You need to tune it for your cluster size.
Now, restart Zeppelin. You should now be able to run:
%spark.pyspark print(mx.__version__) print(np.__version__) 1.6.0 1.14.6
Now you can create GPU context:
ctx = mx.gpu(0)
and it should work as a charm.
So now you have power of Spark — you can easily distribute job and use all slaves. And also, you have GPU at your hand, so whenever you use ndarray
from mxnet, it can use the GPU power.
If you don’t want to use GPU, then just install these libraries instead:
yes | sudo /mnt/home/hadoop/anaconda/bin/pip install --pre mxnet /mnt/home/hadoop/anaconda/bin/conda install -y -c dglteam dgl
and use mx.cpu()
context. This works as well, obviously, much slower. For my use case GPU calculations were 80 times faster than when running on CPU.