Usage ===== Basic Usage ----------- Grab some HTML: >>> import requests >>> html = requests.get('https://www.github.com/').text Then use :func:`formasaurus.extract_forms ` to detect form and field types: >>> import formasaurus >>> formasaurus.extract_forms(html) [(, {'fields': {'q': 'search query'}, 'form': 'search'}), (, {'fields': {'user[email]': 'email', 'user[login]': 'username', 'user[password]': 'password'}, 'form': 'registration'})] .. note:: To detect form and field types Formasaurus needs to train prediction models on user machine. This is done automatically on first call; models are saved to a file and then reused. :func:`formasaurus.extract_forms ` returns a list of (form, info) tuples, one tuple for each ``
`` element on a page. ``form`` is a lxml Element for a form, ``info`` dict contains form and field types. Only fields which are 1. visible to user; 2. have non-empty ``name`` attribute are returned - other fields usually should be either submitted as-is (hidden fields) or not sent to the server at all (fields without ``name`` attribute). There are edge cases like fields filled with JS or fields which are made invisible using CSS, but all bets are off if page uses JS heavily and all we have is HTML source. By default, info dict contains only most likely form and field types. To get probabilities pass ``proba=True``: >>> formasaurus.extract_forms(html, proba=True, threshold=0.05) [(, {'fields': {'q': {'search query': 0.999129068423436}}, 'form': {'search': 0.99580680143321776}}), (, {'fields': {'user[email]': {'email': 0.9980438256540791}, 'user[login]': {'username': 0.9877912041558733}, 'user[password]': {'password': 0.9968113886622333}}, 'form': {'login': 0.12481875549840604, 'registration': 0.86248202363754578}})] Note that Formasaurus is less certain about the second form type - it thinks most likely the form is a registration form (0.86%), but the form also looks similar to a login form (12%). ``threshold`` argument can be used to filter out low-probability options; we used 0.05 in this example. To get probabilities of all classes use ``threshold=0``. If field types are not needed you can speed up processing using ``fields=False`` option. In this case `'fields'` results won't be computed: >>> formasaurus.extract_forms(html, fields=False) [(, {'form': 'search'}), (, {'form': 'registration'})] To extract form and field types from individual form elements use :func:`formasaurus.classify ` or :func:`formasaurus.classify_proba `. They accept lxml Elements. Let's load an HTML file using lxml: >>> import lxml.html >>> tree = lxml.html.parse("http://google.com") and then classify the first form on this page: >>> form = tree.xpath('//form')[0] >>> formasaurus.classify(form) {'fields': {'btnG': 'submit button', 'btnI': 'submit button', 'q': 'search query'}, 'form': 'search'} >>> formasaurus.classify_proba(form, threshold=0.1) {'fields': {'btnG': {'submit button': 0.9254039698573596}, 'btnI': {'submit button': 0.9642014575642849}, 'q': {'search query': 0.9959819637966439}}, 'form': {'search': 0.98794025545508202}} ``fields=False`` arguments works here as well: >>> formasaurus.classify_proba(form, threshold=0.1, fields=False) {'form': {'search': 0.98794025545508202}} In this example the data is loaded from an URL; of course, data may be loaded from a local file or from an in-memory object, or you may already have the tree loaded (e.g. with Scrapy). Form Types ---------- Formasaurus detects these form types:: precision recall f1-score support search 0.91 0.96 0.93 415 login 0.97 0.96 0.96 246 registration 0.95 0.88 0.91 165 password/login recovery 0.88 0.84 0.86 105 contact/comment 0.87 0.94 0.91 138 join mailing list 0.87 0.89 0.88 132 order/add to cart 0.94 0.64 0.76 74 other 0.66 0.69 0.68 143 avg / total 0.89 0.89 0.89 1418 88.9% forms are classified correctly. Quality is estimated based on cross-validation results: all annotated data is split into 20 folds, then model is trained on 19 folds and tries to predict form types in the remaining fold. This is repeated to get predictions for the whole dataset. See also: https://en.wikipedia.org/wiki/Precision_and_recall Field Types ----------- By deafult, Formasaurus detects these field types: * ``username`` * ``password`` * ``password confirmation`` - "enter the same password again" * ``email`` * ``email confirmation`` - "enter the same email again" * ``username or email`` - a field where both username and email are accepted * ``captcha`` - image captcha or a puzzle to solve * ``honeypot`` - this field usually should be left blank * ``TOS confirmation`` - "I agree with Terms of Service", "I agree to follow website rules", "It is OK to process my personal info", etc. * ``receive emails confirmation`` - a checkbox which means "yes, it is ok to send me some sort of emails" * ``remember me checkbox`` - common on login forms * ``submit button`` - a button user should click to submit this form * ``cancel button`` * ``reset/clear button`` * ``first name`` * ``last name`` * ``middle name`` * ``full name`` * ``organization name`` * ``gender`` * ``day`` * ``month`` * ``year`` * ``full date`` * ``time zone`` * ``DST`` - Daylight saving time preference * ``country`` * ``city`` * ``state`` * ``address`` - other address information * ``postal code`` * ``phone`` - phone number or its part * ``fax`` * ``url`` * ``OpenID`` * ``about me text`` * ``comment text`` * ``comment title or subject`` * ``security question`` - "mother's maiden name" * ``answer to security question`` * ``search query`` * ``search category / refinement`` - search parameter, filtering option * ``product quantity`` * ``style select`` - style/theme select, common on forums * ``sorting option`` - asc/desc order, items per page * ``other number`` * ``other read-only`` - field with information user shouldn't change * all other fields are classified as ``other``. Quality estimates (based on 20-fold cross-validation):: precision recall f1-score support username 0.82 0.91 0.86 202 password 1.00 0.99 0.99 368 password confirmation 0.98 0.99 0.99 103 email 0.94 0.97 0.96 615 email confirmation 0.96 0.82 0.88 28 username or email 0.75 0.33 0.46 36 captcha 0.81 0.81 0.81 96 honeypot 0.83 0.34 0.49 29 TOS confirmation 0.88 0.51 0.65 84 receive emails confirmation 0.35 0.57 0.43 87 remember me checkbox 0.96 1.00 0.98 119 submit button 0.94 0.98 0.96 380 cancel button 0.83 0.50 0.62 10 reset/clear button 1.00 0.83 0.91 12 first name 0.89 0.86 0.88 102 last name 0.87 0.85 0.86 101 middle name 1.00 0.57 0.73 7 full name 0.74 0.80 0.77 136 organization name 0.74 0.44 0.55 32 gender 0.95 0.81 0.88 75 time zone 1.00 0.71 0.83 7 DST 1.00 1.00 1.00 5 country 0.89 0.81 0.85 52 city 0.95 0.68 0.80 57 state 0.97 0.69 0.81 42 address 0.76 0.70 0.73 93 postal code 0.97 0.83 0.89 82 phone 0.83 0.84 0.83 110 fax 1.00 1.00 1.00 9 url 0.92 0.68 0.78 34 OpenID 1.00 0.75 0.86 4 about me text 0.62 0.38 0.48 13 comment text 0.88 0.91 0.90 135 comment title or subject 0.68 0.47 0.56 129 security question 0.67 0.22 0.33 9 answer to security question 0.67 0.29 0.40 7 search query 0.90 0.95 0.92 385 search category / refinement 0.92 0.94 0.93 518 product quantity 0.98 0.81 0.88 62 style select 0.94 1.00 0.97 15 sorting option 0.92 0.63 0.75 35 other number 0.32 0.24 0.27 34 full date 0.61 0.61 0.61 23 day 0.90 0.76 0.83 25 month 0.92 0.81 0.86 27 year 0.96 0.79 0.87 34 other read-only 0.91 0.36 0.51 28 other 0.66 0.77 0.71 773 avg / total 0.85 0.85 0.84 5369 84.5% fields are classified correctly. All fields are classified correctly in 76.1% forms.