AAAI Publications, Twenty-Fourth AAAI Conference on Artificial Intelligence

Font Size: 
Learning to Surface Deep Web Content
Zhaohui Wu, Lu Jiang, Qinghua Zheng, Jun Liu

Last modified: 2010-07-05


We propose a novel deep web crawling framework based on reinforcement learning. The crawler is regarded as an agent and deep web database as the environment. The agent perceives its current state and submits a selected action (query) to the environment according to Q-value. Based on the framework we develop an adaptive crawling method. Experimental results show that it outperforms the state of art methods in crawling capability and breaks through the assumption of full-text search implied by existing methods.


hidden web; deep web crawling; reinforcement learning

Full Text: PDF