You can use the ssldump utility to examine, decrypt, and decode SSL-encrypted packet streams managed by the BIG-IP system. The ssldump utility can act on packet streams real-time as they traverse the system, or on a packet capture file saved in the libpcap format, such as that produced by the tcpdump utility. Although it is possible for the ssldump utility to decode and display live traffic real-time as it traverses the BIG-IP system, it is rarely the most effective method to examine the voluminous and complex output of the ssldump utility. Capturing the target traffic to a file using the tcpdump utility, then decoding the file using the ssldump utility offers a better opportunity to examine the traffic in detail.

Here are the steps

1. Capture the traffic
2. Examine the SSL handshake and other SSL traffic
3. Examine the decrypted application data

Capture the traffic

Capture traffic that contains the SSL traffic you want to examine.

When you capture SSL conversations for ssldump examination, follow these guidelines:

• If you use a browser to test, first close all existing browser windows and then use a newly-opened browser window to reproduce the issue to ensure a new session key is used. The ssldump utility cannot decrypt traffic for which the handshake including the key exchange was not seen.
• To write the captured packets to a file for examination with the ssldump utility, you must specify the -w option with the name of the file to which the captured data should be stored.
• Use the -i option to specify the interface or VLAN from which traffic is to be captured.
• Use the appropriate tcpdump filters to include only the traffic you want to examine.
• If you want to decrypt and examine the application data, you must capture the entire packet by specifying a value of 0 or the maximum size of the target packet to the -s option.
• Consider using the -v (verbose) option to increase the level of detail captured.

Capturing traffic examples

if you want to save for examination client-side traffic to a specific SSL virtual server listening on the VLAN external, the following command includes the appropriate options and filters on the virtual server's IP address and port:

tcpdump -vvv -s 0 -nni external -w /var/tmp/www-ssl-client.cap host 10.1.1.100 and port 443

If you want to examine server-side traffic from one client to any pool member, use the -i option to specify the VLAN on which the servers reside, and filter on the client IP address, the server subnet, and the port on which the servers are listening. To do so, use the following command:

tcpdump -vvv -s 0 -nni internal -w /var/tmp/www-ssl-server.cap host 192.168.22.33 and net 10.1.1.0/24 and port 8080

The traffic matching the specified filter is saved to the indicated capture file.

The options used are:

• -vvv Maximum verbosity
• -s Snaplength (0 captures full packets)
• -nn Do not resolve host or service names
• -i Interface - can be ifname or vlan name
• -w Write output to file

 

Examine the SSL handshake and other SSL traffic

SSL connections are established on top of an existing TCP connection using an SSL handshake that accomplishes the following:

• The client and server negotiate security capabilities, such as the public-key algorithm, the symmetric key algorithm, and compression algorithms.
• The server transmits its certificate to the client, allowing the client to validate the identity of the server.
• The client and server exchange session key information.
• The client may also send its certificate to the server, allowing the server to validate the identity of the client.

The handshake transactions consist of a number of SSL record messages. These messages can be examined by executing the ssldump utility using the -r option to specify the path and name of the tcpdump capture file to be examined. Other useful options include the following:

• -n Do not resolve host names.
• -A Print all fields (ssldump, by default, prints only the most interesting).
• -e Print absolute timestamps.
• -d Display application data, including traffic before session initiates.
• -M Output a pre-master secret log file (v. 11.2.0 and later)

For example, the following command displays all of the SSL record messages found in the tcpdump capture file named www-ssl-client.cap:

ssldump -nr /var/tmp/www-ssl-client.cap

The SSL records printed by the ssldump utility appear similar to the following example:

New TCP connection #2: 172.16.31.22(32866) <-> 192.168.1.8(8389)
2 1 0.0002 (0.0002) C>S Handshake
ClientHello
Version 3.0
resume [32]=
a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
cipher suites
SSL_DHE_RSA_WITH_AES_256_CBC_SHA
SSL_DHE_DSS_WITH_AES_256_CBC_SHA
SSL_RSA_WITH_AES_256_CBC_SHA
SSL_DHE_RSA_WITH_AES_128_CBC_SHA
SSL_DHE_DSS_WITH_AES_128_CBC_SHA
SSL_RSA_WITH_RC4_128_MD5
SSL_RSA_WITH_RC4_128_SHA
SSL_RSA_WITH_AES_128_CBC_SHA
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Unknown value 0xfeff
SSL_RSA_WITH_3DES_EDE_CBC_SHA
SSL_DHE_RSA_WITH_DES_CBC_SHA
SSL_DHE_DSS_WITH_DES_CBC_SHA
Unknown value 0xfefe
SSL_RSA_WITH_DES_CBC_SHA
SSL_RSA_EXPORT1024_WITH_RC4_56_SHA
SSL_RSA_EXPORT1024_WITH_DES_CBC_SHA
SSL_RSA_EXPORT_WITH_RC4_40_MD5
SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5
compression methods
NULL
2 2 0.0277 (0.0274) S>C Handshake
ServerHello
Version 3.0
session_id[32]=
a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
cipherSuite SSL_RSA_WITH_3DES_EDE_CBC_SHA
compressionMethod NULL
2 3 0.0277 (0.0000) S>C ChangeCipherSpec
2 4 0.0277 (0.0000) S>C Handshake
2 5 0.0282 (0.0005) C>S ChangeCipherSpec
2 6 0.0282 (0.0000) C>S Handshake
2 7 0.0282 (0.0000) C>S application_data
2 8 0.0289 (0.0006) S>C application_data
2 9 0.0289 (0.0000) S>C application_data
2 10 0.0292 (0.0003) C>S application_data
2 11 0.0296 (0.0003) S>C application_data
2 12 0.0296 (0.0000) S>C application_data
1 24 3.5016 (0.5372) S>C application_data
1 25 3.5016 (0.0000) S>C application_data
2 13 0.5424 (0.5128) C>S application_data
2 14 0.5429 (0.0005) S>C application_data
2 15 0.5429 (0.0000) S>C application_data
1 26 6.0378 (2.5362) C>S application_data
1 27 6.0411 (0.0033) S>C application_data
1 28 6.0411 (0.0000) S>C application_data
2 16 3.1243 (2.5814) C>S application_data
2 17 3.1455 (0.0212) S>C application_data
2 18 3.1455 (0.0000) S>C application_data
1 29 9.2325 (3.1914) C>S application_data
1 30 9.2359 (0.0033) S>C application_data
1 31 9.2359 (0.0000) S>C application_data
1 32 9.3185 (0.0826) C>S application_data
2 19 6.3589 (3.2133) C>S application_data
1 33 9.3276 (0.0090) S>C application_data
1 34 9.3276 (0.0000) S>C application_data
2 20 6.3632 (0.0043) S>C application_data
2 21 6.3632 (0.0000) S>C application_data
1 35 12.3565 (3.0289) C>S application_data
1 36 12.3682 (0.0116) S>C application_data
1 37 12.3682 (0.0000) S>C application_data

The first line defines a new TCP connection which appears similar to the following example:

New TCP connection #2: 172.16.31.22(32866) <-> 192.168.1.8(8080)

To differentiate records belonging to different connections, each connection is numbered. The example defines connection 2. The host that sends the first SYN is printed on the left and the host that responds is printed on the right. In most cases, the SSL client is printed on the left with the SSL server on the right. In this case we have a connection from 172.16.31.22 port 32866 to 192.168.1.8 port 8080.

Subsequent lines represent SSL records sent between the client and the server. The printout of each SSL record begins with a record line. It contains the connection number with which the record is associated, and the sequence number of the record itself, followed by two time stamps. The first time stamp is the time, in seconds, since the beginning of the connection. The second time stamp is the time, in seconds, since the previous record on the same connection. The next column indicates the origin of the message. Communications originating from the client are indicated in the next column by C>S (client to server), while messages originating from the server are marked with S>C (server to client). The last column indicates the type of SSL record message that was received, which can be either Handshake, Alert, ChangeCipherSpec, or application_data. Finally, the ssldump utility may print record-specific data on the rest of the line. For Handshake records, the utility prints the handshake message. For application_data messages, the utility prints the decrypted application data.

By default, the ssldump utility decodes and displays useful details of some SSL record messages.

For example:

• ClientHello

  Details the version, offered cipher suites, and session id, if provided.

• ServerHello

  Details the version, session_id, chosen cipher suite, and compression method.

• Alert

  Details type and level, if provided.

The following example is the ssldump output for the second record on connection 2:

2 2 0.0277 (0.0274) S>C Handshake
ServerHello
Version 3.0
session_id[32]=
a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
cipherSuite SSL_RSA_WITH_3DES_EDE_CBC_SHA
compressionMethod NULL

The record was sent by the server, and it is a Handshake record that contains a Server Hello message.

 

Examine the decrypted application data

Quote

Decrypting the SSL application data may expose sensitive information such as credit card numbers and passwords

The ssldump utility must have access to either the (asymmetric) private key from the server you want to debug, or the (symmetric) pre-master secret keys. Using the pre-master secret keys allows you to examine the decrypted application data in Wireshark without having access to the (asymmetric) private key.

Examine the decrypted application data using the (symmetric) pre-master secret keys

Beginning in BIG-IP 11.2.0, the ssldump -M option allows you to create a pre-master secret (PMS) key log file. You can load the PMS log file into Wireshark (1.6 and later) along with the capture file and use it to decrypt the application data without having access to the server's private key. This option gives F5 Support the ability to fully decrypt sessions in the targeted capture file without revealing sensitive information about the private key.

To run ssldump using the -M option to create a pre-master secret key log file, perform the following procedure:

1. Log in to the BIG-IP command line.
2. Capture the packet trace containing the SSL traffic (refer to the Capturing the target traffic section).
3. To create a pre-master secret key log file, use the following ssldump syntax:

   ssldump -r /path/to/capture_file -k /path/to/private_key -M /path/to/pre-master-key_log_file

   For example, the following ssldump command reads the www-ssl-client1.cap capture file using the test.org key file to decrypt the session, and creates the PMS log file called client1.pms:

   ssldump -r /var/tmp/www-ssl-client1.cap -k /config/filestore/files_d/Common_d/certificate_key_d/\:Common\:test.org.key_1 -M /var/tmp/client1.pms

   You can now load the capture file and the PMS log file into Wireshark (1.6 and later).

Note: To load the pre-master secret (PMS) key log file in Wireshark, go to Edit > Preferences > Protocols > TLS, and enter the path and file name of the PMS key in the (Pre)-Master-Secret log filename field.

Examine the decrypted application data using the (asymmetric) private key

To decrypt and display application data, the ssldump utility will need a copy of the private key from the server you want to debug. If you do not have the key, the application data cannot be decrypted, and you will only be able to examine and decode the SSL handshake packets.

Quote

Not all ciphers provide the ability to decrypt SSL traffic using a utility such as ssldump. Depending on the cipher negotiated, the ssldump utility may not be able to derive enough information from the SSL handshake and the server’s private key to decrypt the application data. Examples of such SSL ciphers would be the Diffie-Hellman Ephemeral (DHE) cipher suites and export-grade RSA cipher suites. If you have to decrypt application data for a virtual server, from BIG-IP 11.6.0, you can use an iRule to get a premaster key for the DHE/ECDHE cipher suite. For details, refer to K16700: Decrypting SSL traffic using the SSL::sessionsecret iRule command. If you are using software earlier than BIG-IP 11.6.0, you can change the cipher suite used in your SSL profile so traffic can be decrypted with ssldump. To do so, make a note of the cipher string currently configured in the SSL profile, then temporarily modify the SSL profile to specify a custom cipher string such as NONE:AES128-SHA.

In the previous example, the ssldump command that is provided prints the application_data record type but does not display the application data itself. Since no key was provided, the application data has not been decrypted. To print the decrypted application data, use the -k option to specify the path and name of the file containing the server's private key.

For example:

ssldump -Aed -nr /var/tmp/www-ssl-client.cap -k /config/ssl/ssl.key/www-ssl.key

 

Decoded application data records printed by ssldump appear similar to the following example:

3 7 1247932735.6286 (0.0031) C>SV3.1(436) application_data
---------------------------------------------------------------
GET / HTTP/1.1
Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; .NET CLR 1.1.4322; msn OptimizedIE8;ENUS)
Accept-Encoding: gzip, deflate
Host: 172.24.72.169
Connection: Keep-Alive

3 8 1247932735.8015 (0.1728) S>CV3.1(1540) application_data
---------------------------------------------------------------
HTTP/1.1 200 OK
Date: Sat, 18 Jul 2009 15:25:37 GMT
Server: Apache/2.2.3 (Debian) PHP/4.4.4-8+etch4 mod_ssl/2.2.3 OpenSSL/0.9.8g mod_perl/2.0.2 Perl/v5.8.8
Last-Modified: Tue, 08 Jan 2008 22:59:06 GMT
ETag: "3306ee-8be-ec750280"
Accept-Ranges: bytes
Content-Length: 2238
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

Locate a BIG-IP virtual server's private key

The private key that the BIG-IP system uses for a virtual server is specified in the Client SSL profile applied to the virtual server. The key file locations are listed below:

BIG-IP 11.x - 16.x:

/config/filestore/files_d/<partition_name>_d/certificate_key_d/

For example:

/config/filestore/files_d/Common_d/certificate_key_d/:Common:test.org.key_1

To determine the key used for a particular virtual server, examine the virtual server configuration to determine the name of the SSL profiles applied to the virtual server, then examine the SSL profile configuration to determine the name of the key file.

 

Resumed TLS handshake

One of the most common reasons an ssldump may not decrypt application data, is if the data is contained within a resumed TLS session. Public key operations are expensive in terms of processing power during the initial setup and key exchange. TLS specifications allow a secure shortcut by using the session_id to resume an SSL connection for which the key exchange was already performed.

In situations where the SSL communication is using a resumed session, ssldump will not be able decrypt the application data unless the capture file contains the initial handshake containing the asymmetric key exchange and session_id. The ssldump utility relies on the information that is exchanged during the initial session setup to decrypt the data.

The following packet capture output example shows a resumed connection:

A client requesting to resume an SSL session appears similar to the following example:

New TCP connection #2: 192.0.2.42(37811) <-> 198.51.100.47(443)
2 1 1296947622.6312 (0.0006) C>SV3.1(156) Handshake
ClientHello
Version 3.1
random[32]=
4d 4d d9 b3 c1 db aa 8e 2a fc 95 ce c0 66 dc 49
a9 49 a5 bf bc c3 3d 67 e5 ba be 77 44 44 6f 8b
resume [32]=
c4 44 ea 86 e2 ba f5 40 4b 44 b4 c2 3a d8 b4 ad
4c dc 13 0d 6c 48 f2 70 19 c3 05 f4 06 e5 ab a9
cipher suites
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
Unknown value 0x45
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Unknown value 0x88
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
Unknown value 0x44
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Unknown value 0x87
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_RC4_128_SHA
Unknown value 0x90
Unknown value 0x91
Unknown value 0x8f
Unknown value 0x8e
TLS_RSA_WITH_AES_128_CBC_SHA
Unknown value 0x41
TLS_RSA_WITH_AES_256_CBC_SHA
Unknown value 0x84
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
Unknown value 0x8c
Unknown value 0x8d
Unknown value 0x8b
Unknown value 0x8a
compression methods
NULL

A server agreeing to resume the session returns the ServerHello with the same session_id as displayed below:

2 2 1296947622.6313 (0.0001) S>CV3.1(74) Handshake
ServerHello
Version 3.1
random[32]=
19 21 d7 55 c1 14 65 63 54 23 62 b7 c4 30 a2 f0
b8 c4 20 06 86 ed 9c 1f 9e 46 0f 42 79 45 8a 29
session_id[32]=
c4 44 ea 86 e2 ba f5 40 4b 44 b4 c2 3a d8 b4 ad
4c dc 13 0d 6c 48 f2 70 19 c3 05 f4 06 e5 ab a9
cipherSuite TLS_RSA_WITH_RC4_128_SHA
compressionMethod NULL

The previous example is a resumed session; the client sends a ClientHello that includes a resume[], the server responds with a ServerHello that contains the same session_id the client sent.

The server is under no obligation to resume a session. This is specified within the TLS 1.0 specification RFC2246:

session_id
This is the identity of the session corresponding to this
connection. If the ClientHello.session_id was non-empty, the
server will look in its session cache for a match. If a match is
found and the server is willing to establish the new connection
using the specified session state, the server will respond with
the same value as was supplied by the client. This indicates a
resumed session and dictates that the parties must proceed
directly to the finished messages. Otherwise this field will
contain a different value identifying the new session. The server
may return an empty session_id to indicate that the session will
not be cached and therefore cannot be resumed. If a session is
resumed, it must be resumed using the same cipher suite it was
originally negotiated with.

To avoid this situation, you can use one of the following methods:

• Temporarily disable the SSL session cache in the Client SSL profile by disabling the Renegotiation option. Disabling the SSL session cache causes the BIG-IP system to perform a full SSL handshake for each connection.
• Force the client to start a new session.

Here is a nice video / tutorial that you may also find helpful

And this video might be helpful as well