How to Debug Applications with Strace, Python, and Wireshark

As a full-stack developer, debugging is an essential skill that you‘ll use on a daily basis. Whether you‘re working on a complex web application, a data processing pipeline, or a machine learning model, bugs are an inevitable part of the development process. The key to being an effective debugger is having a robust toolkit and knowing when and how to apply each tool for maximum impact.

In this in-depth guide, we‘ll explore three powerful debugging tools that every developer should have in their arsenal: strace for debugging system calls and process interactions, Python‘s built-in pdb debugger for interactive debugging of Python scripts, and Wireshark for capturing and analyzing network traffic. We‘ll discuss common debugging scenarios, walk through detailed examples, and share expert tips and best practices to help you become a more efficient and effective debugger. Let‘s dive in!

Debugging System Calls with Strace

strace is a powerful command-line tool for Linux that allows you to trace the system calls and signals made by a running process. This can be incredibly useful for diagnosing issues related to file I/O, process communication, and resource usage.

To use strace, simply prepend it to the command you want to debug:

strace ls -l

This will output a detailed log of all the system calls made by the ls command, including the arguments passed to each call, the return values, and any errors encountered.

One common use case for strace is debugging file permission issues. For example, let‘s say you have a Python script that tries to read from a file, but fails with a PermissionError:

with open(‘data.txt‘, ‘r‘) as f:
    data = f.read()

You can use strace to see which system calls are failing and why:

strace python script.py

In the strace output, look for lines containing open() or access() calls, which will show you the file path and the requested permissions:

access("data.txt", R_OK)                = -1 EACCES (Permission denied)
open("data.txt", O_RDONLY)              = -1 EACCES (Permission denied)

This tells you that the script is trying to open data.txt for reading, but doesn‘t have the necessary permissions. You can then use ls -l data.txt to check the file permissions and modify them with chmod if needed.

Another useful application of strace is debugging issues with child processes or process communication. For instance, if you have a script that spawns a child process using subprocess.Popen(), you can use strace to see how the two processes are interacting:

import subprocess

proc = subprocess.Popen([‘ls‘, ‘-l‘], stdout=subprocess.PIPE)
output = proc.communicate()[0]

By running this script with strace, you can see the fork(), execve(), and pipe() system calls used to create the child process and communicate with it:

fork()                                  = 12345
execve("/bin/ls", ["ls", "-l"], ...)    = 0
pipe([3, 4])                            = 0
read(3, "total 0\n-rw-r--r-- 1 user user 0 Apr 29 10:00 data.txt\n", 4096) = 57
write(1, "total 0\n-rw-r--r-- 1 user user 0 Apr 29 10:00 data.txt\n", 57) = 57

This output shows the creation of the child process (fork and execve), the creation of a pipe for inter-process communication (pipe), and the reading and writing of data through that pipe (read and write). If there were any issues with these system calls (e.g. a missing executable, a broken pipe, etc.), strace would help you identify and diagnose them.

Interactive Debugging with Python‘s PDB

When it comes to debugging Python scripts, one of the most powerful tools at your disposal is the built-in Python Debugger (pdb). With pdb, you can pause execution of your script at any point, step through your code line by line, inspect variables and objects, and even modify the program state on the fly.

To use pdb, simply import it at the top of your script and add a pdb.set_trace() call wherever you want to pause execution and enter the debugger:

import pdb

def complex_function(arg1, arg2):
    # Do some complex calculation
    result = arg1 + arg2
    pdb.set_trace()
    return result

When you run the script and it reaches the pdb.set_trace() line, execution will pause and you‘ll be dropped into the pdb interactive shell:

> /path/to/script.py(6)complex_function()
-> result = arg1 + arg2
(Pdb)

From here, you can use various pdb commands to inspect and control the program execution:

• n (next): Execute the current line and move to the next line
• s (step): Step into the current function call (if any)
• c (continue): Continue execution until the next breakpoint or the end of the program
• p (print): Print the value of an expression or variable
• l (list): List the source code around the current line
• u (up) and d (down): Move up or down the call stack
• h (help): Show a list of available commands

For example, let‘s say you‘re debugging a complex function that takes two arguments and returns their sum, but you‘re getting unexpected results. You can add a breakpoint inside the function and use pdb to inspect the arguments and the result:

(Pdb) p arg1
42
(Pdb) p arg2
‘hello‘
(Pdb) n
TypeError: unsupported operand type(s) for +: ‘int‘ and ‘str‘

Aha! The issue is that arg2 is a string, but the function expects a number. You can fix the bug by converting arg2 to an integer before passing it to the function.

pdb also allows you to set conditional breakpoints that only trigger when a certain expression is true. This can be useful for debugging issues that only occur under specific conditions or with certain inputs. To set a conditional breakpoint, use the break command followed by the file name, line number, and condition:

import pdb

def calculate_average(numbers):
    total = sum(numbers)
    count = len(numbers)
    pdb.set_trace()
    return total / count

(Pdb) break calculate_average, count == 0
Breakpoint 1 at /path/to/script.py:5
(Pdb) c
> /path/to/script.py(5)calculate_average()
-> count = len(numbers)
(Pdb) p numbers
[]
(Pdb) n
ZeroDivisionError: division by zero

In this example, the conditional breakpoint only triggers when count is zero, which helps us identify the issue with dividing by zero when the input list is empty.

By combining pdb with judicious use of logging statements and other debugging techniques, you can quickly diagnose and fix even the most stubborn bugs in your Python code.

Analyzing Network Traffic with Wireshark

Wireshark is a powerful open-source tool for capturing, filtering, and analyzing network traffic. It allows you to inspect the data flowing in and out of your applications at a granular level, which can be invaluable for debugging network-related issues.

To use Wireshark for debugging, start by launching the application and selecting the network interface you want to capture traffic from (e.g. Wi-Fi, Ethernet, localhost). Then, start the capture and perform the action you want to debug, such as making an API request or loading a web page. Once you‘ve captured the relevant traffic, stop the capture and begin analyzing the results.

Wireshark captures are displayed in a table view, with each row representing a single packet. Clicking on a packet reveals its details in the panes below, including the raw bytes, protocol layers, and any application-layer data. You can use Wireshark‘s powerful display filters to narrow down the captured traffic and find the packets you‘re interested in.

For example, let‘s say you‘re debugging an issue with a RESTful API integration in your Python application. You suspect that the API server is returning an error response, but you‘re not sure why. You can use Wireshark to capture the HTTP traffic between your application and the API server and inspect the request and response data.

To start, configure your application to use an HTTP proxy (e.g. Burp Suite or Charles Proxy) so that you can capture the traffic in Wireshark. Then, start a new capture and make a request to the API endpoint. Once the request is complete, stop the capture and filter the traffic using the http display filter.

In the packet list, find the HTTP request sent by your application and select it. In the packet details pane, expand the Hypertext Transfer Protocol section to view the request headers and body. Check that the request method, URL, headers, and payload match your expectations.

Next, find the corresponding HTTP response packet and examine the status code, headers, and body. If the response contains an error message, this will help you identify the issue. For example, you might see a 400 Bad Request status code and an error message indicating that a required parameter is missing or invalid.

HTTP/1.1 400 Bad Request
Content-Type: application/json
{"error": "Missing required parameter ‘api_key‘"}

With this information, you can modify your application code to include the missing parameter and retry the request.

Wireshark can also be useful for debugging more complex network issues, such as TCP connection problems, SSL/TLS handshake failures, or DNS resolution errors. By using display filters like tcp.flags, ssl, or dns, you can isolate the relevant packets and diagnose issues at the protocol level.

For instance, if you‘re investigating a problem with TCP connections timing out, you can use the tcp.analysis.flags filter to show only packets with TCP flags indicating a problem, such as tcp.analysis.retransmission or tcp.analysis.duplicate_ack. This can help you identify network congestion, misconfigured firewalls, or other issues affecting TCP traffic.

Conclusion

Debugging is an essential skill for any developer, but it can be challenging and time-consuming without the right tools and techniques. In this guide, we‘ve explored three powerful debugging tools that can help you diagnose and fix issues quickly and efficiently:

• strace for tracing system calls and debugging process interactions
• Python‘s built-in pdb debugger for interactive debugging of Python scripts
• Wireshark for capturing and analyzing network traffic

By mastering these tools and applying them in a systematic and strategic way, you‘ll be able to tackle even the trickiest bugs with confidence and ease. Remember to:

1. Reproduce the issue consistently: Before you start debugging, make sure you can reproduce the problem reliably. This will make it easier to test your hypotheses and verify that your fixes actually work.

2. Gather information: Use tools like strace, pdb, and Wireshark to gather as much information as possible about the problem. Look for error messages, unexpected behavior, or performance bottlenecks.

3. Form hypotheses: Based on the information you‘ve gathered, form one or more hypotheses about the root cause of the issue. Try to be as specific as possible, and prioritize the most likely explanations.

4. Test your hypotheses: Use your debugging tools to test each hypothesis in turn. Modify the code or configuration, add breakpoints or logging statements, or capture additional traffic as needed.

5. Iterate and refine: If your initial hypotheses don‘t pan out, don‘t be discouraged. Use the new information you‘ve gathered to refine your hypotheses and continue testing until you find the root cause.

6. Fix the problem: Once you‘ve identified the root cause, implement a fix and test it thoroughly to make sure it resolves the issue without introducing new bugs.

7. Document and share: Finally, document your findings and share them with your team. This will help everyone learn from the experience and avoid similar issues in the future.

By following these steps and leveraging the power of strace, pdb, and Wireshark, you‘ll be able to debug any application with skill and confidence. Happy debugging!