Difference between revisions of "Talk:SMILA/Documentation/HowTo/How to implement a crawler"
(New page: Comment originally by Ivan Churkin: In my oppinion implementing crawler interface and creating service is not good idea for 3rd party developers. Its required to have too much knowlege ab...) |
m (Talk:SMILA/Development Guidelines/How to implement a crawler moved to Talk:SMILA/Documentation/HowTo/How to implement a crawler) |
||
| (3 intermediate revisions by 2 users not shown) | |||
| Line 16: | Line 16: | ||
And it will be written one wrapper class that will implement Crawler interface and it will use "DataExtractor" user's object for crawling(creating Record and so on). | And it will be written one wrapper class that will implement Crawler interface and it will use "DataExtractor" user's object for crawling(creating Record and so on). | ||
and plz look at the page comment | and plz look at the page comment | ||
| + | |||
| + | |||
| + | === Develop your crawler === | ||
| + | that section is, to say the least, insufficient. | ||
| + | looking into already existing crawlers doesn't help much, since there's not much comment but some rather tangled code. | ||
| + | a clear description of the methods to implement and what they are supposed to do (outlining how-to, maybe) is the minimum of required information. | ||
| + | i've browsed the SMILA wiki a lot lately and am still suprised how a project aiming that high can provide so little information suitable for starters. | ||
Latest revision as of 09:29, 24 January 2012
Comment originally by Ivan Churkin:
In my oppinion implementing crawler interface and creating service is not good idea for 3rd party developers. Its required to have too much knowlege about technologies (SCA, declarative services, osgi, our interfaces...).
Its much better to suggest for implementation some simple iterateable interface like
interface DataExtractor { void start(IndexOrderConfiruration config); boolean moveNext(); Object readAttribute(String name); void finish(); }
And it will be written one wrapper class that will implement Crawler interface and it will use "DataExtractor" user's object for crawling(creating Record and so on). and plz look at the page comment
Develop your crawler
that section is, to say the least, insufficient. looking into already existing crawlers doesn't help much, since there's not much comment but some rather tangled code. a clear description of the methods to implement and what they are supposed to do (outlining how-to, maybe) is the minimum of required information. i've browsed the SMILA wiki a lot lately and am still suprised how a project aiming that high can provide so little information suitable for starters.