Java爬蟲實現Jsoup利用dom方法遍歷Document對象
先給出網頁地址:
https://wall.alphacoders.com/featured.php?lang=Chinese
主要步驟:
利用Jsoup的connect方法獲取Document對象
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese"; Document doc = Jsoup.connect(html).get();
內容過長,就不再顯示。
我們以這部分為例:
<ul class="nav nav-pills"> <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li> <li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li> <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li> <li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li> <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li> <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li> <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li> <li><a id="change_consent">Change Consent</a></li> </ul>
我們先找到所有的ul:
Elements elements = doc.getElementsByTag("ul");
輸出如下:
<ul class="nav navbar-nav center"> <li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li> <li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美獎品</a> </li> </ul> <ul class="nav navbar-nav navbar-right center"> <li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="https://static.alphacoders.com/wa/Chinese-flag.png" alt="Chinese-flag"> 中文 </a> </li> <li> <a href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登錄</a> </li> <li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 註冊</a> </li> </ul> <ul class="pagination"> <li class="active"><a id="prev_page" href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一頁</a></li> <li class="active"><a>1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=8" rel="external nofollow" >8</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=9" rel="external nofollow" >9</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=10" rel="external nofollow" >10</a></li> <li><a>...</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li> <li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一頁 ></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一頁</a></li> <li class="active"><a>1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> <li><a>...</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一頁 ></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ><< </a></li> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一頁</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一頁 ></a></li> <li><a title="末頁 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" > >></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> </ul> <ul class="nav nav-pills"> <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li> <li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li> <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li> <li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li> <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li> <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li> <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li> <li><a id="change_consent">Change Consent</a></li> </ul>
可以發現class為"nav nav-pills"的隻有一個,我們找到它:
Elements elements = doc.getElementsByTag("ul"); //System.out.println(elements); Element tempElement = null; for(Element element : elements) { if (element.className().equals("nav nav-pills")) { tempElement = element; //System.out.println(element.className()); break; } }
循環遍歷這個ul,輸出其中每一個li裡每一個a的href和rel屬性:
Elements li = tempElement.getElementsByTag("li"); for(Element element : li) { Elements element2 = element.getElementsByTag("a"); for(Element element3 : element2) { String hrefString = element3.attr("href"); String relString = element3.attr("rel"); if(hrefString != "" && relString != "") { System.out.println("href="%20+%20hrefString%20+%20" rel="external nofollow" rel="external nofollow" " + "rel=" + relString); } } }
最終結果:
href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow
完整代碼:
import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; import org.jsoup.select.Elements; import java.io.IOException; import org.jsoup.Jsoup; /** * @ClassName: Jsoup_Test * @description: * @author: KI * @Date: 2020年8月17日 下午8:15:14 */ public class Jsoup_Test { public static void main(String[] args) throws IOException { // TODO 自動生成的方法存根 String html = "https://wall.alphacoders.com/featured.php?lang=Chinese"; Document doc = Jsoup.connect(html).get(); System.out.println(doc); Elements elements = doc.getElementsByTag("ul"); //System.out.println(elements); Element tempElement = null; for(Element element : elements) { if (element.className().equals("nav nav-pills")) { tempElement = element; //System.out.println(element.className()); break; } } System.out.println(tempElement); Elements li = tempElement.getElementsByTag("li"); for(Element element : li) { Elements element2 = element.getElementsByTag("a"); for(Element element3 : element2) { String hrefString = element3.attr("href"); String relString = element3.attr("rel"); if(hrefString != "" && relString != "") { System.out.println("href="%20+%20hrefString%20+%20" rel="external nofollow" rel="external nofollow" " + "rel=" + relString); } } } } }
到此這篇關於Java爬蟲實現Jsoup利用dom方法遍歷Document對象的文章就介紹到這瞭,更多相關Java Jsoup遍歷Document對象內容請搜索WalkonNet以前的文章或繼續瀏覽下面的相關文章希望大傢以後多多支持WalkonNet!
推薦閱讀:
- java爬蟲jsoup解析HTML的工具學習
- Java 獲取網站圖片的示例代碼
- Java使用httpRequest+Jsoup爬取紅藍球號碼
- java爬蟲Jsoup主要類及功能使用詳解
- jsoup 框架的使用小結