I just wanted to point out that “deep web search technology” has already been developed. Take a look at http://www.scienceresearch.com/, http://www.worldwidescience.org/, http://www.mednar.com/, and http://www.scitopia.org/. (I could go on). My company, http://www.deepwebtech.com/, is responsible for many such websites (public and private).

It’s actually quite difficult for a number of reasons, the biggest being that most deep web content is proprietary, requiring credentials or a subscription to access. Another component of deep web content is publicly available information that’s hidden behind a form. For example, to get at an article, you need to fill out a form. Automated search tools don’t necessarily know how to fill out a form, without basic assumptions.

This means, until ALL publishers create common interfaces to their information (as well as allow free searching, even if the content itself requires a subscription), the deep web will always be difficult to reach.

Larry.