added vpn ip rotation
This commit is contained in:
419
TROUBLESHOOTING_DE.md
Normal file
419
TROUBLESHOOTING_DE.md
Normal file
@@ -0,0 +1,419 @@
|
||||
# ProtonVPN-Integration: Troubleshooting & FAQ
|
||||
|
||||
## Inhaltsverzeichnis
|
||||
- [Häufige Probleme](#häufige-probleme)
|
||||
- [Konfiguration Debug](#konfiguration-debug)
|
||||
- [Extension-Selektoren aktualisieren](#extension-selektoren-aktualisieren)
|
||||
- [Performance-Tipps](#performance-tipps)
|
||||
- [Testing ohne VPN](#testing-ohne-vpn)
|
||||
|
||||
---
|
||||
|
||||
## Häufige Probleme
|
||||
|
||||
### Problem 1: Extension wird nicht gefunden
|
||||
**Symptom:** `Failed to navigate to ProtonVPN extension popup`
|
||||
|
||||
**Ursache:**
|
||||
- Extension nicht installiert
|
||||
- Falsche Extension-ID in Konfiguration
|
||||
- Chrome lädt Extension nicht automatisch
|
||||
|
||||
**Lösung:**
|
||||
```bash
|
||||
# 1. Extension ID überprüfen
|
||||
# Chrome öffnen → chrome://extensions/ → ProtonVPN Details anklicken
|
||||
# Extension ID kopieren und in .env eintragen
|
||||
|
||||
PROTONVPN_EXTENSION_ID=ghmbeldphafepmbegfdlkpapadhbakde # Aktualisieren!
|
||||
|
||||
# 2. Manuell in Chrome installieren
|
||||
# https://chrome.google.com/webstore/detail/protonvpn/ghmbeldphafepmbegfdlkpapadhbakde
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem 2: "Disconnect button not found" oder "Connect button not found"
|
||||
**Symptom:** Extension-Buttons werden nicht gefunden
|
||||
|
||||
**Ursache:**
|
||||
- Extension UI hat sich geändert (Update)
|
||||
- XPath-Selektoren sind veraltet
|
||||
- HTML-Struktur unterscheidet sich zwischen Browser-Versionen
|
||||
|
||||
**Lösung:**
|
||||
```rust
|
||||
// 1. Browser DevTools öffnen
|
||||
// Chrome: F12 → Öffne chrome-extension://[ID]/popup.html
|
||||
|
||||
// 2. HTML inspizieren:
|
||||
// Right-click auf Button → Inspect Element
|
||||
|
||||
// 3. XPath-Selektoren aktualisieren
|
||||
// In src/scraper/protonvpn_extension.rs:
|
||||
//
|
||||
// Falls neuer HTML-Struktur, z.B.:
|
||||
// <button class="vpn-connect-btn">Connect</button>
|
||||
//
|
||||
// Neuer XPath:
|
||||
let xpath = "//button[@class='vpn-connect-btn']";
|
||||
|
||||
// Oder alternative Strategien hinzufügen zur find_and_click_button()-Funktion
|
||||
```
|
||||
|
||||
**Modifizierte find_and_click_button() für neue Selektoren:**
|
||||
|
||||
```rust
|
||||
async fn find_and_click_button(&self, client: &Client, text: &str) -> Result<()> {
|
||||
let lower_text = text.to_lowercase();
|
||||
|
||||
let xpath_strategies = vec![
|
||||
// Text-basiert (case-insensitive)
|
||||
format!(
|
||||
"//button[contains(translate(text(), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), '{}')]",
|
||||
lower_text
|
||||
),
|
||||
// CSS-Klassen (AnpassEN nach Bedarf)
|
||||
format!("//button[contains(@class, '{}')]", text),
|
||||
// Data-Attribute
|
||||
format!("//*[@data-action='{}']", lower_text),
|
||||
// Aria-Label
|
||||
format!("//*[@aria-label='{}']", text),
|
||||
// SVG + Text (für moderne UIs)
|
||||
format!("//*[contains(., '{}')][@role='button']", text),
|
||||
];
|
||||
|
||||
for xpath in xpath_strategies {
|
||||
if let Ok(element) = client.find(fantoccini::LocatorStrategy::XPath(&xpath)).await {
|
||||
element.click().await?;
|
||||
debug!("Clicked: {}", text);
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
|
||||
Err(anyhow!("Button '{}' not found", text))
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem 3: VPN verbindet sich nicht oder Timeout
|
||||
**Symptom:** `Failed to connect to ProtonVPN server 'US' within 15 seconds`
|
||||
|
||||
**Ursachen:**
|
||||
1. ProtonVPN-Server überlastet
|
||||
2. Netzwerk-Latenz
|
||||
3. Falsche Server-Name
|
||||
4. Browser-Erweiterung nicht vollständig geladen
|
||||
|
||||
**Lösungen:**
|
||||
|
||||
**A. Timeout erhöhen:**
|
||||
```rust
|
||||
// In protonvpn_extension.rs, connect_to_server():
|
||||
// Erhöhe von 30 auf 60 Versuche
|
||||
for attempt in 0..60 { // 30s → 60 Versuche = 30s timeout
|
||||
sleep(Duration::from_millis(500)).await;
|
||||
if self.is_connected(client).await.unwrap_or(false) {
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**B. Server-Namen überprüfen:**
|
||||
```bash
|
||||
# Gültige ProtonVPN-Server (für Free-Tier):
|
||||
# US, UK, JP, NL, etc.
|
||||
#
|
||||
# Oder mit Nummern:
|
||||
# US-Free#1, US-Free#2, UK-Free#1
|
||||
# US#1, US#2 (für Plus-Tier)
|
||||
|
||||
# In .env überprüfen:
|
||||
VPN_SERVERS=US,UK,JP,NL
|
||||
# NICHT: VPN_SERVERS=US-Free#1, UK-Free#1 (zu viele Leerzeichen)
|
||||
```
|
||||
|
||||
**C. Extension-Status überprüfen:**
|
||||
```rust
|
||||
// Debug: Printe HTML vor Connect-Versuch
|
||||
let extension_url = format!("chrome-extension://{}/popup.html", self.extension_id);
|
||||
client.goto(&extension_url).await?;
|
||||
sleep(Duration::from_secs(1)).await;
|
||||
|
||||
let html = client.source().await?;
|
||||
println!("=== EXTENSION HTML ===");
|
||||
println!("{}", html);
|
||||
println!("=====================");
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem 4: IP-Adresse wird nicht extrahiert
|
||||
**Symptom:** `Failed to extract IP from whatismyipaddress.com`
|
||||
|
||||
**Ursache:** HTML-Struktur hat sich geändert
|
||||
|
||||
**Lösung:**
|
||||
```rust
|
||||
// In protonvpn_extension.rs, get_current_ip():
|
||||
// Füge Debug-Ausgabe hinzu:
|
||||
|
||||
let page_source = client.source().await?;
|
||||
println!("=== PAGE SOURCE ===");
|
||||
println!("{}", page_source);
|
||||
println!("===================");
|
||||
|
||||
// Dann neue Regex/Extraction-Logik basierend auf aktuellem HTML
|
||||
```
|
||||
|
||||
**Alternative IP-Check-Services:**
|
||||
```rust
|
||||
// icanhazip.com (gibt nur IP zurück)
|
||||
client.goto("https://icanhazip.com/").await?;
|
||||
sleep(Duration::from_secs(1)).await;
|
||||
let ip = client.source().await?.trim().to_string();
|
||||
|
||||
// ifconfig.me
|
||||
client.goto("https://ifconfig.me/").await?;
|
||||
sleep(Duration::from_secs(1)).await;
|
||||
let ip = client.source().await?.trim().to_string();
|
||||
|
||||
// checkip.amazonaws.com
|
||||
client.goto("https://checkip.amazonaws.com/").await?;
|
||||
sleep(Duration::from_secs(1)).await;
|
||||
let ip = client.source().await?.trim().to_string();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem 5: Session-Manager erstellt Sessions, aber VPN verbindet nicht
|
||||
**Symptom:** `VPN session created, but is_connected() returns false`
|
||||
|
||||
**Ursache:**
|
||||
- WebDriver-Client hat Extension nicht geladen
|
||||
- ChromeDriver-Instanz verwirrt zwischen mehreren Sessions
|
||||
|
||||
**Lösung:**
|
||||
|
||||
Sicherstellen, dass jeder WebDriver-Client die Extension hat:
|
||||
|
||||
```rust
|
||||
// In webdriver.rs, ChromeInstance::new() oder new_with_extension():
|
||||
// Extension-Pfad muss zu Chrome-Start mitgegeben werden
|
||||
|
||||
let mut cmd = Command::new("chromedriver-win64/chromedriver.exe");
|
||||
cmd.arg("--port=0");
|
||||
|
||||
// Hinweis: Extension wird automatisch geladen, wenn in Chrome installiert
|
||||
// Für Testing kann man auch Headless-Modus deaktivieren:
|
||||
// cmd.arg("--headless=false"); // Damit man Browser sieht
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Konfiguration Debug
|
||||
|
||||
### Enable Debug Logging
|
||||
```bash
|
||||
# Terminal
|
||||
RUST_LOG=debug cargo run
|
||||
|
||||
# Oder in code:
|
||||
tracing_subscriber::fmt()
|
||||
.with_max_level(tracing::Level::DEBUG) // Statt INFO
|
||||
.init();
|
||||
```
|
||||
|
||||
### Überprüfen Sie die geladene Konfiguration
|
||||
```bash
|
||||
# .env Datei überprüfen
|
||||
cat .env
|
||||
|
||||
# Oder Ausgabe am Start ansehen
|
||||
cargo run
|
||||
|
||||
# Output sollte zeigen:
|
||||
# ✓ Config loaded | VPN: enabled | Max Parallel: 3
|
||||
```
|
||||
|
||||
### Test-Konfigurationen
|
||||
|
||||
**Minimal (ohne VPN):**
|
||||
```env
|
||||
ENABLE_VPN_ROTATION=false
|
||||
MAX_PARALLEL_TASKS=1
|
||||
```
|
||||
|
||||
**Mit VPN, aber langsam:**
|
||||
```env
|
||||
ENABLE_VPN_ROTATION=true
|
||||
VPN_SERVERS=US,UK
|
||||
TASKS_PER_VPN_SESSION=5
|
||||
MAX_PARALLEL_TASKS=1 # Nur eine Instanz für Testing
|
||||
RUST_LOG=debug
|
||||
```
|
||||
|
||||
**Mit VPN, normal:**
|
||||
```env
|
||||
ENABLE_VPN_ROTATION=true
|
||||
VPN_SERVERS=US,UK,JP,NL,DE
|
||||
TASKS_PER_VPN_SESSION=10
|
||||
MAX_PARALLEL_TASKS=3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Extension-Selektoren aktualisieren
|
||||
|
||||
### Wie man neue Selektoren findet
|
||||
|
||||
1. **Chrome öffnen:**
|
||||
```
|
||||
chrome://extensions/ → ProtonVPN → Details
|
||||
```
|
||||
|
||||
2. **Popup öffnen:**
|
||||
```
|
||||
Navigate to: chrome-extension://[ID]/popup.html
|
||||
```
|
||||
|
||||
3. **DevTools öffnen (F12):**
|
||||
- Elements Tab
|
||||
- Inspect Element (Button rechts oben)
|
||||
- Klicke auf Button im Popup
|
||||
|
||||
4. **HTML kopieren:**
|
||||
```html
|
||||
<!-- Beispiel neuer Button -->
|
||||
<button class="btn btn-primary" id="connect-btn">
|
||||
<i class="icon-vpn"></i>
|
||||
Connect
|
||||
</button>
|
||||
```
|
||||
|
||||
5. **Neuen XPath erstellen:**
|
||||
```rust
|
||||
// Option 1: Nach ID
|
||||
"//button[@id='connect-btn']"
|
||||
|
||||
// Option 2: Nach Klasse
|
||||
"//button[@class='btn btn-primary']"
|
||||
|
||||
// Option 3: Nach Text
|
||||
"//button[contains(text(), 'Connect')]"
|
||||
```
|
||||
|
||||
6. **In find_and_click_button() hinzufügen:**
|
||||
```rust
|
||||
let xpath_strategies = vec![
|
||||
"//button[@id='connect-btn']".to_string(),
|
||||
"//button[@class='btn btn-primary']".to_string(),
|
||||
// ... other strategies
|
||||
];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance-Tipps
|
||||
|
||||
### 1. Batch-Processing statt paralleles Threading
|
||||
```rust
|
||||
// ❌ LANGSAM: Zu viele parallele Instances
|
||||
let pool = ChromeDriverPool::new(10).await?;
|
||||
|
||||
// ✅ SCHNELLER: Weniger Instances, mehr Tasks pro Instance
|
||||
let pool = ChromeDriverPool::new(3).await?;
|
||||
config.max_tasks_per_instance = 20; // Recycel nach 20 Tasks
|
||||
```
|
||||
|
||||
### 2. VPN-Verbindung optimieren
|
||||
```rust
|
||||
// ❌ LANGSAM: Jeder Task rotiert IP
|
||||
TASKS_PER_VPN_SESSION=1
|
||||
|
||||
// ✅ SCHNELLER: Mehrere Tasks pro IP
|
||||
TASKS_PER_VPN_SESSION=10
|
||||
```
|
||||
|
||||
### 3. Timing anpassen
|
||||
```rust
|
||||
// Zu aggressive:
|
||||
sleep(Duration::from_millis(100)).await;
|
||||
|
||||
// Besser (für VPN):
|
||||
sleep(Duration::from_millis(500)).await;
|
||||
|
||||
// Für Disconnect/Connect Sequenzen:
|
||||
// Mindestens 2-3 Sekunden zwischen Operationen
|
||||
```
|
||||
|
||||
### 4. Server-Auswahl
|
||||
```env
|
||||
# ❌ Problematic: Zu viele ähnliche Server
|
||||
VPN_SERVERS=US-Free#1,US-Free#2,US-Free#3,US-Free#4
|
||||
|
||||
# ✅ Better: Mix aus verschiedenen Ländern
|
||||
VPN_SERVERS=US-Free#1,UK-Free#1,JP-Free#1,NL-Free#1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing ohne VPN
|
||||
|
||||
### 1. VPN deaktivieren für Testing
|
||||
```env
|
||||
ENABLE_VPN_ROTATION=false
|
||||
MAX_PARALLEL_TASKS=1
|
||||
ECONOMIC_LOOKAHEAD_MONTHS=1 # Kleinere Datenmenge
|
||||
```
|
||||
|
||||
### 2. Mock-Tests schreiben
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_vpn_session_manager() {
|
||||
let mgr = VpnSessionManager::new(
|
||||
vec!["US".to_string(), "UK".to_string()],
|
||||
3
|
||||
);
|
||||
|
||||
mgr.create_new_session().await.unwrap();
|
||||
assert!(mgr.get_current_session().await.is_some());
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Extension-Fehler isolieren
|
||||
```bash
|
||||
# Test nur extension.rs
|
||||
cargo test --lib scraper::protonvpn_extension
|
||||
```
|
||||
|
||||
### 4. Scraping ohne VPN testen
|
||||
```bash
|
||||
# Setze ENABLE_VPN_ROTATION=false
|
||||
ENABLE_VPN_ROTATION=false RUST_LOG=info cargo run
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Weitere Ressourcen
|
||||
|
||||
- **ProtonVPN Chrome Extension:** https://chrome.google.com/webstore/detail/protonvpn/ghmbeldphafepmbegfdlkpapadhbakde
|
||||
- **Fantoccini (WebDriver):** https://docs.rs/fantoccini/latest/fantoccini/
|
||||
- **Tokio Runtime:** https://tokio.rs/
|
||||
- **Tracing/Logging:** https://docs.rs/tracing/latest/tracing/
|
||||
|
||||
---
|
||||
|
||||
## Support & Debugging-Checkliste
|
||||
|
||||
Bevor Sie ein Issue öffnen:
|
||||
|
||||
- [ ] `.env` ist korrekt konfiguriert
|
||||
- [ ] ProtonVPN Extension ist installiert
|
||||
- [ ] Chrome + ChromeDriver sind kompatibel
|
||||
- [ ] `RUST_LOG=debug` wurde ausgeführt um Logs zu sehen
|
||||
- [ ] Selektoren wurden mit Browser DevTools überprüft
|
||||
- [ ] Test ohne VPN (`ENABLE_VPN_ROTATION=false`) funktioniert
|
||||
- [ ] Server-Namen sind korrekt (z.B. `US`, nicht `USA`)
|
||||
|
||||
Reference in New Issue
Block a user