Repository navigation
Fix Bing fallback in -g returning no links - #6150
Merged
Merged
Conversation
While looking into sqlmapproject#6009 I noticed the Bing fallback in -g never returns anything, even when Bing itself serves a normal results page. Two things changed on Bing's side. The result anchor now carries target="_blank" before href, so '<h2><a href="..." h=' no longer matches at all. And the href is no longer the target URL but a tracking redirect (https://www.bing.com/ck/a?...&u=a1<base64url>), where the real link sits base64url-encoded after the "a1" prefix in the u parameter. The regex now allows extra attributes on both tags and captures either the u payload of a ck/a redirect or a plain href. The payload is decoded with decodeBase64(), which already deals with the URL-safe alphabet and the missing padding. Plain hrefs still work the same as before (with & unescaped), in case Bing serves the old markup to some clients. Checked against a saved live results page (0 -> 10 links) and the old markup, and --smoke-test passes. Google still can't be scraped since it only serves a JS page now, so this doesn't fully close sqlmapproject#6009, but Bing is usable again as a fallback when DuckDuckGo blocks the IP.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #6009.
While looking into #6009 I tried each engine in
lib/utils/search.pyon its own. DuckDuckGo works, Google only serves the JS page now, and Bing returns a normal results page but sqlmap pulls 0 links out of it.The Bing markup has changed in two ways:
<h2 ...><a target="_blank" href="...">, so the old<h2><a href="..." h=pattern doesn't match anymore.https://www.bing.com/ck/a?...&u=a1<base64url>redirect, and the real link is the base64url part aftera1.BING_REGEXnow allows extra attributes and captures either theupayload of ack/alink or a plain href. The payload goes through the existingdecodeBase64(), which already handles the URL-safe alphabet and the missing padding. Plain hrefs work the same as before (with&unescaped), in case some clients still get the old markup.Testing:
python sqlmap.py --smoke-testpassesThis doesn't fully fix #6009, since Google can't be scraped without JS, but it gives
-ga working fallback again when DuckDuckGo starts throwing its anomaly page (which happened to me after a few requests while testing).One side note: DDG's html endpoint seems to return 10 results per page now (its next-page form posts
s=10), while we use(gpage - 1) * 30for the offset. I couldn't verify that properly because DDG blocked me mid-testing, so I've left it out of this PR.