sudo yum update -y --releasever=latest --cve CVE-2019-0232

sudo shutdown -r now

It should fix the mentioned hole but after bringing up the new cluster and running

[hadoop@ip-10-0-1-84 ~]$ yum info tomcat8
Loaded plugins: priorities, update-motd, upgrade-helper
6 packages excluded due to repository priority protections
Installed Packages
Name        : tomcat8
Arch        : noarch
Version     : 8.5.29
Release     : 1.77.amzn1
Size        : 324 k
Repo        : installed
From repo   : amzn-updates
Summary     : Apache Servlet/JSP Engine, RI for Servlet 3.1/JSP 2.3 API
URL         : http://tomcat.apache.org/ 
License     : ASL 2.0
Description : Tomcat is the servlet container that is used in the official Reference
            : Implementation for the Java Servlet and JavaServer Pages technologies.
            : The Java Servlet and JavaServer Pages specifications are developed by
            : Sun under the Java Community Process.
            :
            : Tomcat is developed in an open and participatory environment and
            : released under the Apache Software License version 2.0. Tomcat is intended
            : to be a collaboration of the best-of-breed developers from around the world.

we can see that it installs version 8.5.29 of tomcat (instead of 8.5.42 or something newer). Why? Let’s SSH into the EMR and check:

[hadoop@ip-10-0-1-84 ~]$ yum --showduplicates list tomcat8
Loaded plugins: priorities, update-motd, upgrade-helper
6 packages excluded due to repository priority protections
Installed Packages
tomcat8.noarch                                                                                                8.5.29-1.77.amzn1                                                                                                 @amzn-updates
Available Packages
tomcat8.noarch                                                                                                8.5.16-1.74.amzn1                                                                                                 amzn-main
tomcat8.noarch                                                                                                8.5.23-1.75.amzn1                                                                                                 amzn-updates
tomcat8.noarch                                                                                                8.5.28-1.76.amzn1                                                                                                 amzn-updates
tomcat8.noarch                                                                                                8.5.29-1.77.amzn1                                                                                                 amzn-updates

So there is no newer version. But if we try applying the CVE patch again we get:

[hadoop@ip-10-0-1-84 /]$ sudo yum update -y --releasever=latest --cve CVE-2019-0232
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                                                                                                                                                                             | 2.1 kB  00:00:00
amzn-updates                                                                                                                                                                                                          | 2.5 kB  00:00:00
emr-applications                                                                                                                                                                                                      | 2.9 kB  00:00:00
8 packages excluded due to repository priority protections
6 package(s) needed (+0 related) for security, out of 130 available
Resolving Dependencies
--> Running transaction check
---> Package tomcat8.noarch 0:8.5.29-1.77.amzn1 will be updated
---> Package tomcat8.noarch 0:8.5.42-1.80.amzn1 will be an update
---> Package tomcat8-el-3.0-api.noarch 0:8.5.29-1.77.amzn1 will be updated
---> Package tomcat8-el-3.0-api.noarch 0:8.5.42-1.80.amzn1 will be an update
---> Package tomcat8-jsp-2.3-api.noarch 0:8.5.29-1.77.amzn1 will be updated
---> Package tomcat8-jsp-2.3-api.noarch 0:8.5.42-1.80.amzn1 will be an update
---> Package tomcat8-lib.noarch 0:8.5.29-1.77.amzn1 will be updated
---> Package tomcat8-lib.noarch 0:8.5.42-1.80.amzn1 will be an update
---> Package tomcat8-servlet-3.1-api.noarch 0:8.5.29-1.77.amzn1 will be updated
---> Package tomcat8-servlet-3.1-api.noarch 0:8.5.42-1.80.amzn1 will be an update
---> Package tomcat8-webapps.noarch 0:8.5.29-1.77.amzn1 will be updated
---> Package tomcat8-webapps.noarch 0:8.5.42-1.80.amzn1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================
 Package                                                           Arch                                             Version                                                     Repository                                              Size
=============================================================================================================================================================================================================================================
Updating:
 tomcat8                                                           noarch                                           8.5.42-1.80.amzn1                                           amzn-updates                                           102 k
 tomcat8-el-3.0-api                                                noarch                                           8.5.42-1.80.amzn1                                           amzn-updates                                           110 k
 tomcat8-jsp-2.3-api                                               noarch                                           8.5.42-1.80.amzn1                                           amzn-updates                                            68 k
 tomcat8-lib                                                       noarch                                           8.5.42-1.80.amzn1                                           amzn-updates                                           4.3 M
 tomcat8-servlet-3.1-api                                           noarch                                           8.5.42-1.80.amzn1                                           amzn-updates                                           255 k
 tomcat8-webapps                                                   noarch                                           8.5.42-1.80.amzn1                                           amzn-updates                                           358 k

Transaction Summary
=============================================================================================================================================================================================================================================
Upgrade  6 Packages

Total download size: 5.2 M
Downloading packages:
(1/6): tomcat8-el-3.0-api-8.5.42-1.80.amzn1.noarch.rpm                                                                                                                                                                | 110 kB  00:00:00
(2/6): tomcat8-8.5.42-1.80.amzn1.noarch.rpm                                                                                                                                                                           | 102 kB  00:00:00
(3/6): tomcat8-jsp-2.3-api-8.5.42-1.80.amzn1.noarch.rpm                                                                                                                                                               |  68 kB  00:00:00
(4/6): tomcat8-webapps-8.5.42-1.80.amzn1.noarch.rpm                                                                                                                                                                   | 358 kB  00:00:00
(5/6): tomcat8-lib-8.5.42-1.80.amzn1.noarch.rpm                                                                                                                                                                       | 4.3 MB  00:00:00
(6/6): tomcat8-servlet-3.1-api-8.5.42-1.80.amzn1.noarch.rpm                                                                                                                                                           | 255 kB  00:00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                        6.0 MB/s | 5.2 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : tomcat8-servlet-3.1-api-8.5.42-1.80.amzn1.noarch                                                                                                                                                                         1/12
  Updating   : tomcat8-el-3.0-api-8.5.42-1.80.amzn1.noarch                                                                                                                                                                              2/12
  Updating   : tomcat8-jsp-2.3-api-8.5.42-1.80.amzn1.noarch                                                                                                                                                                             3/12
  Updating   : tomcat8-lib-8.5.42-1.80.amzn1.noarch                                                                                                                                                                                     4/12
  Updating   : tomcat8-8.5.42-1.80.amzn1.noarch                                                                                                                                                                                         5/12
  Updating   : tomcat8-webapps-8.5.42-1.80.amzn1.noarch                                                                                                                                                                                 6/12
  Cleanup    : tomcat8-webapps-8.5.29-1.77.amzn1.noarch                                                                                                                                                                                 7/12
  Cleanup    : tomcat8-8.5.29-1.77.amzn1.noarch                                                                                                                                                                                         8/12
  Cleanup    : tomcat8-lib-8.5.29-1.77.amzn1.noarch                                                                                                                                                                                     9/12
  Cleanup    : tomcat8-jsp-2.3-api-8.5.29-1.77.amzn1.noarch                                                                                                                                                                            10/12
  Cleanup    : tomcat8-el-3.0-api-8.5.29-1.77.amzn1.noarch                                                                                                                                                                             11/12
  Cleanup    : tomcat8-servlet-3.1-api-8.5.29-1.77.amzn1.noarch                                                                                                                                                                        12/12
  Verifying  : tomcat8-webapps-8.5.42-1.80.amzn1.noarch                                                                                                                                                                                 1/12
  Verifying  : tomcat8-jsp-2.3-api-8.5.42-1.80.amzn1.noarch                                                                                                                                                                             2/12
  Verifying  : tomcat8-lib-8.5.42-1.80.amzn1.noarch                                                                                                                                                                                     3/12
  Verifying  : tomcat8-el-3.0-api-8.5.42-1.80.amzn1.noarch                                                                                                                                                                              4/12
  Verifying  : tomcat8-servlet-3.1-api-8.5.42-1.80.amzn1.noarch                                                                                                                                                                         5/12
  Verifying  : tomcat8-8.5.42-1.80.amzn1.noarch                                                                                                                                                                                         6/12
  Verifying  : tomcat8-webapps-8.5.29-1.77.amzn1.noarch                                                                                                                                                                                 7/12
  Verifying  : tomcat8-lib-8.5.29-1.77.amzn1.noarch                                                                                                                                                                                     8/12
  Verifying  : tomcat8-jsp-2.3-api-8.5.29-1.77.amzn1.noarch                                                                                                                                                                             9/12
  Verifying  : tomcat8-8.5.29-1.77.amzn1.noarch                                                                                                                                                                                        10/12
  Verifying  : tomcat8-servlet-3.1-api-8.5.29-1.77.amzn1.noarch                                                                                                                                                                        11/12
  Verifying  : tomcat8-el-3.0-api-8.5.29-1.77.amzn1.noarch                                                                                                                                                                             12/12

Updated:
  tomcat8.noarch 0:8.5.42-1.80.amzn1           tomcat8-el-3.0-api.noarch 0:8.5.42-1.80.amzn1   tomcat8-jsp-2.3-api.noarch 0:8.5.42-1.80.amzn1   tomcat8-lib.noarch 0:8.5.42-1.80.amzn1   tomcat8-servlet-3.1-api.noarch 0:8.5.42-1.80.amzn1
  tomcat8-webapps.noarch 0:8.5.42-1.80.amzn1

Complete!
[hadoop@ip-10-0-1-84 /]$ yum info tomcat8
Loaded plugins: priorities, update-motd, upgrade-helper
6 packages excluded due to repository priority protections
Installed Packages
Name        : tomcat8
Arch        : noarch
Version     : 8.5.42
Release     : 1.80.amzn1
Size        : 326 k
Repo        : installed
From repo   : amzn-updates
Summary     : Apache Servlet/JSP Engine, RI for Servlet 3.1/JSP 2.3 API
URL         : http://tomcat.apache.org/
License     : ASL 2.0
Description : Tomcat is the servlet container that is used in the official Reference
            : Implementation for the Java Servlet and JavaServer Pages technologies.
            : The Java Servlet and JavaServer Pages specifications are developed by
            : Sun under the Java Community Process.
            :
            : Tomcat is developed in an open and participatory environment and
            : released under the Apache Software License version 2.0. Tomcat is intended
            : to be a collaboration of the best-of-breed developers from around the world.

So it was able to update the Tomcat even though the same command run through bootstrap didn’t do it. What’s happening?

Let’s try installing it manually. Add another boostrap action before the AWS script:

#!/bin/sh

set -ex

{
	yum info tomcat8
	yum --showduplicates --releasever=latest list tomcat8
	sudo yum update --releasever=latest --cve CVE-2019-0232
	sudo yum update -y --releasever=latest tomcat8
	sudo yum update -y --releasever=latest tomcat8-8.5.42-1.80.amzn1
} >> /home/hadoop/cves.out 2>> /home/hadoop/cves.err

Logs show this:

Loaded plugins: priorities, update-motd, upgrade-helper
6 packages excluded due to repository priority protections
Available Packages
Name        : tomcat8
Arch        : noarch
Version     : 8.5.29
Release     : 1.77.amzn1
Size        : 102 k
Repo        : amzn-updates/2017.09
Summary     : Apache Servlet/JSP Engine, RI for Servlet 3.1/JSP 2.3 API
URL         : http://tomcat.apache.org/ 
License     : ASL 2.0
Description : Tomcat is the servlet container that is used in the official
            : Reference Implementation for the Java Servlet and JavaServer Pages
            : technologies. The Java Servlet and JavaServer Pages specifications
            : are developed by Sun under the Java Community Process.
            :
            : Tomcat is developed in an open and participatory environment and
            : released under the Apache Software License version 2.0. Tomcat is
            : intended to be a collaboration of the best-of-breed developers
            : from around the world.

Loaded plugins: priorities, update-motd, upgrade-helper
8 packages excluded due to repository priority protections
Available Packages
tomcat8.noarch                  8.5.29-1.77.amzn1                   amzn-main
tomcat8.noarch                  8.5.32-1.78.amzn1                   amzn-updates
tomcat8.noarch                  8.5.40-1.79.amzn1                   amzn-updates
tomcat8.noarch                  8.5.42-1.80.amzn1                   amzn-updates
Loaded plugins: priorities, update-motd, upgrade-helper
8 packages excluded due to repository priority protections
No packages needed for security; 126 packages available
Resolving Dependencies
Loaded plugins: priorities, update-motd, upgrade-helper
8 packages excluded due to repository priority protections
Package(s) tomcat8 available, but not installed.
No packages marked for update
Loaded plugins: priorities, update-motd, upgrade-helper
8 packages excluded due to repository priority protections
Package(s) tomcat8-8.5.42-1.80.amzn1 available, but not installed.
No packages marked for update

So it could find version 8.5.42 but refused to update it. How to fix that? The solution is to install instead of update. Let’s try this:

#!/bin/sh

set -ex

{
	sudo yum install -y --releasever=latest tomcat8-servlet-3.1-api-8.5.42
	sudo yum install -y --releasever=latest tomcat8-el-3.0-api-8.5.42
	sudo yum install -y --releasever=latest tomcat8-jsp-2.3-api-8.5.42
	sudo yum install -y --releasever=latest tomcat8-lib-8.5.42
	sudo yum install -y --releasever=latest tomcat8-8.5.42
	sudo yum install -y --releasever=latest tomcat8-webapps-8.5.42
	exit 0
} >> /home/hadoop/cves.out 2>> /home/hadoop/cves.err

And now it works as expected. Unfortunately, installing specific Tomcat version (instead of latest one) is not the best idea but at least this solves the issue.

But what to do if you have this problem? How to debug it?

First, make sure that your jars are on the classpath. You can add them using spark.driver.extraClassPath (similar for executor) and spark.driver.userClassPathFirst parameters.

Next, verify that your classes are actually loaded. Just add -verbose:class when submitting the job. Next, see the logs and check if the class you want to use is loaded and where it comes from.

Now you should probably see the reason why a method is not found. Typically it is not there (because of SDK version change) or it has slightly different signature. If you still have no clue what’s going on, make sure that you are using right classloader.