Web Reconnaissance Notes

Introduction

Web reconnaissance is the systematic collection of information about target websites/web applications, forming the foundation of security assessments and penetration testing.

Methodology

1. Initial Information Gathering

• Passive Reconnaissance

  • WHOIS lookups

  • DNS enumeration

  • Search engine discovery

  • Certificate transparency logs

  • Web archives analysis

  • Social media research

  • Code repository analysis

• Active Reconnaissance

  • Port scanning

  • Service enumeration

  • Banner grabbing

  • Vulnerability scanning

  • Web spidering

  • Network mapping

2. Domain & DNS Analysis

1. WHOIS Lookup

   whois domain.com

2. DNS Enumeration

   # Basic DNS lookup
   dig domain.com
   
   # MX records
   dig domain.com MX
   
   # Zone transfer attempt
   dig axfr @nameserver domain.com

3. Subdomain Enumeration

   # Using dnsenum
   dnsenum --enum domain.com -f wordlist.txt -r
   
   # Certificate transparency logs
   curl -s "https://crt.sh/?q=domain.com&output=json" | jq -r '.[] | .name_value' | sort -u

3. Virtual Host Discovery

# Using Gobuster
gobuster vhost -u http://target_IP -w wordlist.txt --append-domain

4. Fingerprinting

# Banner grabbing
curl -I domain.com

# WAF detection
wafw00f domain.com

# Comprehensive scan
nikto -h domain.com -Tuning b

5. Web Crawling & Content Discovery

# Using robots.txt
curl domain.com/robots.txt

# Using Scrapy
python3 ReconSpider.py http://domain.com

Essential Tools

1. DNS & Subdomain Tools

• dig: DNS lookup utility

• dnsenum: Comprehensive DNS enumeration

• subfinder: Modern subdomain discovery

• amass: In-depth DNS enumeration and asset discovery

2. Web Scanning Tools

• Nikto: Web server scanner

• Wappalyzer: Technology stack identification

• wafw00f: Web Application Firewall detector

• Burp Suite: Comprehensive web application testing platform

3. Crawling & Content Discovery

• Scrapy: Python-based web crawling framework

• Burp Suite Spider: Web application crawler

• OWASP ZAP: Security testing proxy with crawling capabilities

4. Automation Frameworks

• FinalRecon: All-in-one web reconnaissance tool

• Recon-ng: Web reconnaissance framework

• theHarvester: Email, subdomain, and people gathering tool

• SpiderFoot: OSINT automation tool

Google Dorks for Reconnaissance

Common search operators:

site:domain.com          # Limit to specific domain
inurl:login             # Find login pages
filetype:pdf            # Search for file types
intitle:"index of"      # Directory listing

Well-Known URIs to Check

/.well-known/security.txt
/.well-known/change-password
/.well-known/openid-configuration

Best Practices

1. Start Passive, Then Active

   • Begin with non-intrusive methods

   • Escalate to active scanning only when necessary

2. Document Everything

   • Record all findings

   • Note unusual responses or behavior

   • Track discovered assets

3. Legal & Ethical Considerations

   • Obtain proper authorization

   • Follow scope limitations

   • Respect robots.txt when appropriate

   • Monitor resource usage to avoid DoS

4. Validation & Verification

   • Cross-reference findings from multiple tools

   • Verify discovered vulnerabilities

   • Document false positives

Reporting Tips

1. Organization

   • Group findings by category

   • Prioritize based on risk

   • Include evidence and screenshots

2. Technical Details

   • Document commands used

   • Include tool versions

   • Note any special configurations

3. Remediation

   • Provide clear fixing steps

   • Include references to best practices

   • Suggest testing procedures