Nk_logs_from_timestamps

Home

Description

This is a bash function that searches for logs containing a specified timestamp for a given domain. Here is a breakdown of what the function does:

The function expects two arguments: the domain name and the timestamp to search for.

The function first assigns the domain name and timestamp to the variables $domain and $date_timestamp respectively.

The function then uses another function called gen_log_list() to generate a list of domlogs for the specified domain. This includes both ongoing logs and archived logs.

The matched_logs() function is then called to iterate over each domlog file in the generated list and check if it contains the specified timestamp. If the log file is a gz file, it uses zgrep to search for the timestamp, otherwise it uses grep. If a match is found, the function prints out the path to the log file.

Finally, the function iterates over each log file that contains the specified timestamp and uses either zgrep or grep to search for the timestamp. It then sorts the output by the fourth column and gets only the unique results.

In summary, this function can be used to quickly search for logs containing a specified timestamp for a given domain.

Example

[root@cloudvpsserver]# nk_logs_from_timestamps nkern.net 01/Apr/2023 | tail
142.132.177.174 - - [01/Apr/2023:23:44:06 -0400] "GET /contact-us/ HTTP/1.1" 404 7938 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0"
93.114.185.76 - - [01/Apr/2023:23:44:06 -0400] "GET /wp-login.php HTTP/1.1" 200 5082 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"
142.132.177.174 - - [01/Apr/2023:23:44:07 -0400] "GET / HTTP/1.1" 200 9341 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0"
205.234.21.92 - - [01/Apr/2023:23:47:04 -0400] "GET / HTTP/1.1" 200 52666 "-" "Go-http-client/1.1"
104.131.4.140 - - [01/Apr/2023:23:47:52 -0400] "POST /wp-login.php HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
104.131.4.140 - - [01/Apr/2023:23:47:53 -0400] "GET /wp-login.php HTTP/1.1" 200 5082 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
66.249.65.254 - - [01/Apr/2023:23:56:08 -0400] "GET /images/v3/fast_forward.png HTTP/1.1" 404 7938 "-" "Googlebot-Image/1.0"

Code

nk_logs_from_timestamps() {
# nk_logs_from_timestamps expects both a domain and timestamp.
# If neither are provided exit.
if [ "$1" = "" ] || [ "$2" = "" ]; then
    echo "You must provide a domain and timestamp."
    return 0
fi
# Declare some variables.
domain="$1"
date_timestamp="$2"
user="$(nk_user "$domain")"
archive_dir="$(whodat -l "$domain" | awk '/Archived\ Domlog\ Dir:/ {print $NF}')"

gen_log_list(){
# Generate a list of domlogs.
    # First find archived logs.
    find "$archive_dir" -type f -iname "$domain*.gz"
    # Then find ongoing logs.
    find "/var/log/apache2/domlogs/$user/" -type f -iname "$domain*"
}

matched_logs() {
# Now for every domlog generated by gen_log_list.
for log in $(gen_log_list); do
# If the filename for log ends in .gz then
if [ "$(echo "$log" | grep -Ec ".+\.gz")" != "0" ]; then
    # Set logmatch to the number of hits for the timestamp
    logmatch="$(zgrep -c "$date_timestamp" "$log")"
        # If the number of hits is not zero.
        if [ "$logmatch" != "0" ]; then
            # Print out that we found a matching log.
            echo "$log"
        fi
    return 0
fi
# Otherwise if $log is not a gz file.
# logmatch is equal to the number of hits on the timestamp with normal grep.
logmatch="$(grep -c "$date_timestamp" "$log")"
# If logmatch is not 0 then we have a match.
if [ "$logmatch" != "0" ]; then
    # Print out that we have a matching log.
    echo "$log"
fi
done
}

# Now we can iterate over each log that we found that matches the provided timestamp.
for log in $(matched_logs); do
# If the log is a gz file.
if [ "$(echo "$log" | grep -Ec ".+\.gz")" != "0" ]; then
     # Then grep out the timestamps using zgrep.
     zgrep "$date_timestamp" "$log"
     return 0
fi
# Otherwise just use normal grep.
grep "$date_timestamp" "$log"

done | sort -k 4 | uniq
# Get only the unique results. Since there could be duplicates
# when combining information from live logs and archived.
}