> For the complete documentation index, see [llms.txt](https://edu.noirchapeau.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://edu.noirchapeau.com/web-reconnaissance-notes.md).

# Web Reconnaissance Notes

## Introduction

Web reconnaissance is the systematic collection of information about target websites/web applications, forming the foundation of security assessments and penetration testing.

## Methodology

### 1. Initial Information Gathering

* **Passive Reconnaissance**
  * WHOIS lookups
  * DNS enumeration
  * Search engine discovery
  * Certificate transparency logs
  * Web archives analysis
  * Social media research
  * Code repository analysis
* **Active Reconnaissance**
  * Port scanning
  * Service enumeration
  * Banner grabbing
  * Vulnerability scanning
  * Web spidering
  * Network mapping

### 2. Domain & DNS Analysis

1. **WHOIS Lookup**

   ```bash
   whois domain.com
   ```
2. **DNS Enumeration**

   ```bash
   # Basic DNS lookup
   dig domain.com

   # MX records
   dig domain.com MX

   # Zone transfer attempt
   dig axfr @nameserver domain.com
   ```
3. **Subdomain Enumeration**

   ```bash
   # Using dnsenum
   dnsenum --enum domain.com -f wordlist.txt -r

   # Certificate transparency logs
   curl -s "https://crt.sh/?q=domain.com&output=json" | jq -r '.[] | .name_value' | sort -u
   ```

### 3. Virtual Host Discovery

```bash
# Using Gobuster
gobuster vhost -u http://target_IP -w wordlist.txt --append-domain
```

### 4. Fingerprinting

```bash
# Banner grabbing
curl -I domain.com

# WAF detection
wafw00f domain.com

# Comprehensive scan
nikto -h domain.com -Tuning b
```

### 5. Web Crawling & Content Discovery

```bash
# Using robots.txt
curl domain.com/robots.txt

# Using Scrapy
python3 ReconSpider.py http://domain.com
```

## Essential Tools

### 1. DNS & Subdomain Tools

* **dig**: DNS lookup utility
* **dnsenum**: Comprehensive DNS enumeration
* **subfinder**: Modern subdomain discovery
* **amass**: In-depth DNS enumeration and asset discovery

### 2. Web Scanning Tools

* **Nikto**: Web server scanner
* **Wappalyzer**: Technology stack identification
* **wafw00f**: Web Application Firewall detector
* **Burp Suite**: Comprehensive web application testing platform

### 3. Crawling & Content Discovery

* **Scrapy**: Python-based web crawling framework
* **Burp Suite Spider**: Web application crawler
* **OWASP ZAP**: Security testing proxy with crawling capabilities

### 4. Automation Frameworks

* **FinalRecon**: All-in-one web reconnaissance tool
* **Recon-ng**: Web reconnaissance framework
* **theHarvester**: Email, subdomain, and people gathering tool
* **SpiderFoot**: OSINT automation tool

## Google Dorks for Reconnaissance

Common search operators:

```
site:domain.com          # Limit to specific domain
inurl:login             # Find login pages
filetype:pdf            # Search for file types
intitle:"index of"      # Directory listing
```

## Well-Known URIs to Check

```
/.well-known/security.txt
/.well-known/change-password
/.well-known/openid-configuration
```

## Best Practices

1. **Start Passive, Then Active**
   * Begin with non-intrusive methods
   * Escalate to active scanning only when necessary
2. **Document Everything**
   * Record all findings
   * Note unusual responses or behavior
   * Track discovered assets
3. **Legal & Ethical Considerations**
   * Obtain proper authorization
   * Follow scope limitations
   * Respect robots.txt when appropriate
   * Monitor resource usage to avoid DoS
4. **Validation & Verification**
   * Cross-reference findings from multiple tools
   * Verify discovered vulnerabilities
   * Document false positives

## Reporting Tips

1. **Organization**
   * Group findings by category
   * Prioritize based on risk
   * Include evidence and screenshots
2. **Technical Details**
   * Document commands used
   * Include tool versions
   * Note any special configurations
3. **Remediation**
   * Provide clear fixing steps
   * Include references to best practices
   * Suggest testing procedures
