Java爬蟲實戰(zhàn)指南：按關(guān)鍵字搜索淘寶商品

管理

編輯

刪除

one-Jason 2025-07-09 11:05:36

暫不處理

在電商數(shù)據(jù)分析、市場調(diào)研等領(lǐng)域，獲取淘寶商品信息是一項常見需求。Java 語言憑借其強大的網(wǎng)絡編程能力和豐富的庫支持，成為實現(xiàn)此類爬蟲的理想選擇。本文將詳細介紹如何使用 Java 爬蟲按關(guān)鍵字搜索淘寶商品，并提供完整的代碼示例。

一、準備工作

1. 注冊淘寶開放平臺賬號

在使用淘寶 API 之前，需要在淘寶開放平臺注冊賬號并創(chuàng)建應用。注冊完成后，平臺會分配一個 App Key 和 App Secret，這兩個參數(shù)是調(diào)用 API 時的身份驗證憑證。

2. 添加 Maven 依賴

為了方便地發(fā)送 HTTP 請求和解析 JSON 數(shù)據(jù)，我們需要在項目中添加以下 Maven 依賴：

xml

<dependencies>
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.13</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.10.0</version>
    </dependency>
</dependencies>

二、代碼實現(xiàn)

1. 發(fā)送 HTTP 請求

使用 HttpClient 發(fā)送 GET 請求，調(diào)用淘寶的商品搜索接口：

java

import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import java.io.IOException;
import java.security.MessageDigest;
import java.util.HashMap;
import java.util.Map;

public class TaobaoProductSearchCrawler {
    public static String searchTaobaoProducts(String appKey, String appSecret, String keyword, int page) {
        String timestamp = String.valueOf(System.currentTimeMillis() / 1000);
        Map<String, String> params = new HashMap<>();
        params.put("method", "taobao.item.search");
        params.put("app_key", appKey);
        params.put("timestamp", timestamp);
        params.put("v", "2.0");
        params.put("keyword", keyword);
        params.put("page", String.valueOf(page));
        params.put("sign_method", "md5");
        params.put("sign", generateSignature(appKey, appSecret, timestamp, params));

        try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
            HttpGet request = new HttpGet("https://gw.api.taobao.com/router.json?" + toQueryString(params));
            try (CloseableHttpResponse response = httpClient.execute(request)) {
                if (response.getStatusLine().getStatusCode() == 200) {
                    return EntityUtils.toString(response.getEntity());
                } else {
                    System.out.println("請求失敗，狀態(tài)碼：" + response.getStatusLine().getStatusCode());
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }

    public static String generateSignature(String appKey, String appSecret, String timestamp, Map<String, String> params) {
        StringBuilder signStr = new StringBuilder(appKey + timestamp + appSecret);
        for (Map.Entry<String, String> entry : params.entrySet()) {
            signStr.append(entry.getKey()).append(entry.getValue());
        }
        return md5(signStr.toString());
    }

    public static String md5(String input) {
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            byte[] messageDigest = md.digest(input.getBytes());
            BigInteger no = new BigInteger(1, messageDigest);
            String hashtext = no.toString(16);
            while (hashtext.length() < 32) {
                hashtext = "0" + hashtext;
            }
            return hashtext;
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    public static String toQueryString(Map<String, String> params) {
        StringBuilder sb = new StringBuilder();
        for (Map.Entry<String, String> entry : params.entrySet()) {
            if (sb.length() > 0) {
                sb.append("&");
            }
            sb.append(entry.getKey()).append("=").append(entry.getValue());
        }
        return sb.toString();
    }
}

2. 解析 JSON 響應

使用 Jackson 庫解析返回的 JSON 數(shù)據(jù)，提取商品信息：

java

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

public class Main {
    public static void main(String[] args) {
        String appKey = "your_app_key";
        String appSecret = "your_app_secret";
        String keyword = "iPhone 13";
        int page = 1;
        String response = TaobaoProductSearchCrawler.searchTaobaoProducts(appKey, appSecret, keyword, page);
        if (response != null) {
            parseResponse(response);
        }
    }

    public static void parseResponse(String jsonResponse) {
        try {
            ObjectMapper objectMapper = new ObjectMapper();
            JsonNode rootNode = objectMapper.readTree(jsonResponse);
            int code = rootNode.path("code").asInt();
            String errorMessage = rootNode.path("errorMessage").asText();
            if (code == 0) {
                JsonNode itemsNode = rootNode.path("item_search_response").path("items");
                for (JsonNode itemNode : itemsNode) {
                    String title = itemNode.path("title").asText();
                    String price = itemNode.path("price").asText();
                    String picUrl = itemNode.path("pic_url").asText();
                    String detailUrl = itemNode.path("detail_url").asText();
                    System.out.println("商品標題: " + title);
                    System.out.println("商品價格: " + price);
                    System.out.println("商品圖片: " + picUrl);
                    System.out.println("商品詳情頁 URL: " + detailUrl);
                }
            } else {
                System.out.println("API 請求失敗，錯誤信息: " + errorMessage);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}