Title of Invention

"SYSTEM TO INSTANTLY GENERATE AN ONLINE IMAGE OF A DOCUMENT FROM MULTIPLE IMAGES CAPTURED THROUGH A CAMERA EQUPPED MOBILE DEVICE"

Abstract

A method and system is provided for generating a composite document from part images captured through a mobile camera. A user captures multiple images of a document using the camera of his mobile device, covering all the contents of the document. Once the entire desired content has been captured through the multiple images, all these images are sent to an application server, using suitable communication service. At the application server these individual images are juxtaposed, matched for overlapping regions and stitched to form the original document intended. The document image is then sent either to the mobile phone of the original user or sent to any other business application as desired by the original user.

Full Text

Field of the Invention The instant invention relates to a system and method to generate a composite document from part images captured through a mobile camera. It generally relates to matching and stitching various images to form a document. Background Modern world is witnessing information being processed at incredible speeds. Newer, faster, more efficient means of processing information are constantly hitting the markets and offering business fraternity better value for money. However, it becomes crucial in such a scenario to quickly and accurately provide the input information to be processed at the right place. Such an information-enabling system is becoming critical by the day for businesses to stay competitive. Driven by the dynamic business environment and consistent demands of enterprises and organizations to expand their operations, modern-day executives are constantly on the move. All paper-intensive industries such as Insurance, Banking, Courier, Logistics, Mortgage, Healthcare, Publishing and Manufacturing are always in need of a system to reduce the turnaround time of their activities. For example, insurance companies have their executives visit customers and obtain their details on paper documents and application forms. These application forms and documents containing all the customer information are then taken back to the regional/central head office for further processing. This entire process is cumbersome and time consuming. Insurance companies are always competing against each other in cutting down turn around time for policy issuance. To cut down the turnaround time, faster processing is the keyword. Entering all the information in the database manually is prone to many errors. Also there is always a backlog regarding entering, processing and updating data. US6563948 relates to method of building an electronic file. It relies on sophisticated cameras to capture document images. It involves processing of images locally, either in the camera or to a PC connected to the camera. The matching strings should be relatively long and also occur infrequently in a text file. This is detrimental to a large number of document images being rendered unsuitable for extraction. Further, the camera is busy and unusable for the whole duration when the image stitching and text file generation is being done. US 6,975,434 relates to method and apparatus for scanning oversized documents. It focuses on extending the range of sizes of document, which can be scanned using a flat bed scanner without making major modifications to hardware components of the scanner. This involves costly scanners. US 20030163833 patent application does not talk about image stitching, and only focuses on distributed processing of images captured through mobile-phone camera. A paper on Fast and robust planar registration by early consensus with applications to document stitching by Pilu, M. Isgro, F. Hewlett-Packard Labs., UK; The approach, used in this paper, is very basic and primitive. Mobile image scanner available athttp://ip.fuiitsu.com/group/labs/downloads/en/business/activities/activities-4/fuiitsu-labs-iodev-001 -en.pdf refers to automatic document orientation correction function. It might be able to efficiently and effectively handle only basic problems such as document orientation and rotation. In the paper entitled, "A fast and reliable planar registration method with applications to document stitching", intensity-based normalized cross-correlation technique for feature point matching is used. This is a basic technique, which might not provide good feature points under adverse conditions. In addition, the stitched images might not blend smoothly. Accordingly, an efficient system and method is needed to instantly generate an image of a document without any local processing and substantially reduce the turnaround time. Objectives and Summary of the Instant Invention It is an objective of the instant invention to obviate the above drawbacks and provide a method and system to generate a composite document. It is another objective of the instant invention to decrease the turnaround time for generating a composite document image. A method to instantly generate an online image of a document from multiple images captured through a camera equipped mobile device, said method comprising capturing a plurality of images of said document using said camera equipped mobile device, each of said images having an overlapped portion, indexing and transmitting together said images, pertaining to a document, to an application server, selecting a first image and an adjacent second image from said images at the application server, extracting and obtaining multiple SIFT feature points in a first image and a second image, said first and second images being adjacent, matching said feature points obtained in the first image with the feature points obtained in the second image, aligning said first and second images and stitching said images based on best matching points, repeating the above steps with all the images till a single composite image is generated and, transmitting said single composite image to a desired location. A system to instantly generate an online image of a document from multiple images captured through a camera equipped mobile device, comprising a mobile device with image capturing means for capturing multiple images of a document, an application server for receiving said images from the mobile device, means for extracting and obtaining multiple SIFT feature points in a first image and a second image, said first and second images being adjacent, means for matching said feature points obtained in the first image with the feature points obtained in the second image, means for aligning said first and second images and stitching said images based on best matching points, and transmitting means for transmitting single composite image to a desired location. Brief description of the drawings Figure 1 describes the system of the instant invention. Figure 2 describes the steps of the method of the instant invention. Detailed description of the instant invention The instant invention uses a mobile communication device for implementing the instant invention. Mobile communication devices with image capturing facility are easily available. The instant invention uses this facility to capture various images of a document and process the data or information therein. The composite document created by this method can be an application form with details filled therein, a report, a newsletter, which needs to be saved online. A scanner cannot be available anytime and everywhere. Accordingly, the instant invention provides an anytime anywhere solution. A system is presented here that generates a composite document image by automatically matching and stitching portions of the image captured by the user using mobile phone camera. Figure 1 describes the system of the instant invention. Modern day mobile phones invariably come equipped with a camera. In an embodiment, the instant invention includes a mobile device (10) with image capturing means. This image capturing means is a camera of at least 2.0 MP resolution. A camera with resolution less than 2MP would not capture a legible image. Capturing a single image of a large document that is legible enough by using a mobile camera is difficult because of the limitations of low-resolution of the mobile camera. Accordingly the system captures various images of portions of a document. Any mobile phone with a good resolution image capturing means can be used here. Only extremely powerful professional digital cameras are able to provide satisfactory resolution if the document has to be captured in a single image. However, these cameras are too costly to be considered for general use. The mobile device has receiving and transmitting means and has service installed to communicate with an application server (11) of the service provider using an available communication service, such as MMS facility, any other phone-based protocol, which can be Web-based, wireless, company-developed custom protocol or any such suitable medium. The images (12) captured by the mobile device are sent to the application server for composition and further processing. The application server receives all the portions of the document as individual images (12.1, 12.2, 12.3, 12.4). The instant invention provides means in the application server for image matching. The application server has instructions stored therein and is configured to perform matching and stitching the images of a single document. The application server has instructions for selecting two adjacent images from the images of a document and extracting and obtaining multiple SIFT feature points in the two images. The server has means for matching said feature points obtained in the first image with the feature points obtained in the second image, aligning said first and second images and stitching the images based on best matching points. The server has further instructions of repeating the above steps with all the images till a single composite image is generated. The images are joined together at appropriate points thereby forming a composite image similar to the original document. This composite image is then sent to the desired business system (14). The application server has receiving and transmitting means for the purpose of this invention. In one embodiment, the desired business system may be the mobile phone of the user who had initially sent the individual images. The composite document image may also be sent to any other system such as a central repository of the company, e-mail, fax, etc. The system will find use in various other application areas, such as exchange of handwritten notes, bill processing, whiteboard notes capturing and document verification. Figure 2 describes the steps of the method of performing the instant invention through a flowchart. Modern day mobile phones invariably come equipped with a camera. In one embodiment, the instant invention is used for filled up forms. A user once having obtained all the customer details on the form and other ancillary documents, uses his mobile phone camera to capture various portions of the forms (20). Multiple images encompassing different portions of the form need to be captured because a single image capturing the entire document is hard to decipher because of the relatively low resolution of the camera available in mobile phones. The user captures the individual image of different portions with some overlap with neighboring portions, which is needed for correct image matching and stitching, otherwise automatic matching will not be possible. The images are captured part-by-part in such a manner that these images combined together capture the entire content of the document. The images of each page of the form are captured. The user captures the images in an order and indexes them (21) so that images are processed and stitched in order at the application server. The mobile device has transmitting and receiving means to send these images to the application server It may, however, be noted that the images pertaining to only one page of a form should be sent in one batch. The application server receives all the portions of the document as individual images. Once the individual part images are received, two adjacent images are taken (22) at a time to be joined together. The adjacency of images is detected based on their indexes. In one embodiment, the server detects the relevant position of all the individual images for a composite image, without the user requiring to index and specify the relevant positioning. Scale Invariant Feature Transforms (SIFT) feature points are extracted from the two images by using gaussian scale space (23). Maximum possible features points are extracted for better results. SIFT image features are not affected by scalar or geometric transformations. These features are located at the local extrema of difference of Gaussian Scale Space. In order to choose better feature points, the extrema in the local neighborhood are considered by a particular threshold. Through iterative thresholding, the value of threshold is constantly changed at run time until sufficient number of feature points is obtained. SIFT Features obtained in one image are matched (24) with those obtained in the other image using K-d trees. This might result into correct as well as false matching points. For further obtaining the best matching feature points, Slope Refinement technique is used. Here, a slope M is determined such that the sum of deviations of slopes between corresponding feature points in two images is minimum. The slope M corresponds to correct matching points. All other matching points with slopes within a threshold of slope M are retained and others rejected. Matching points are further refined using RANSAC (Random Sample Consensus) algorithm for outliers rejection. Using Adaptive Non-Maximal Suppression (ANMS), for image transformation, spatially distributed matching points are selected. Projective transformation is done to align the two images to make them ready for stitching (25) by correction in rotation, scaling, shearing, dilations, etc. Images are stitched based on the best matching point, which are also spatially well distributed so as to cover the entire overlapped area. Finally, gradient blending is done for smooth transitioning of the edges present in the area where the two images are stitched. The method is repeated till all the images of a document at the server are stitched to form a single composite document image. This composite image is then sent to the required destination (26). It may be the mobile phone of the user who had initially sent the individual images or any other system such as a central repository of the company, e-mail, fax, etc. The entire process would substantially reduce the time taken in rendering the availability of information for processing. It will readily be appreciated by those skilled in the art that the present invention is not limited to the specific embodiments shown herein. Thus variations may be made within the scope and spirit of the accompanying claims without sacrificing the principal advantages of the invention. We claim: - 1. A method to instantly generate an online image of a document from multiple images captured through a camera equipped mobile device, said method comprising: capturing a plurality of images of said document using said camera equipped mobile device, each of said images having an overlapped portion, indexing and transmitting together said images, pertaining to a document, to an application server, selecting a first image and an adjacent second image from said images at the application server, extracting and obtaining multiple SIFT feature points in a first image and a second image, said first and second images being adjacent, matching said feature points obtained in the first image with the feature points obtained in the second image aligning said first and second images and stitching said images based on best matching points repeating the above steps with all the images till a single composite image is generated and, transmitting said single composite image to a desired location. 2. The method as claimed in claim 1, wherein said captured images cover different portions of said document. 3. The method as claimed in claim 1, wherein images of the entire content of the document are captured. 4. The method as claimed in claim 1, wherein selecting the first and second adjacent images is done using the indexes of the captured images. 5. The method as claimed in claim 1, wherein said extracting and obtaining multiple SIFT feature points comprises generating Gaussian scale space. 6. The method as claimed in claim 1, wherein said matching feature points comprises using K-d trees and Slope Refinement technique. 7. The method as claimed in claim 1, wherein said aligning and stitching images comprises using Adaptive Non-Maximal Suppression (ANMS) for image transformation. 8. The method as claimed in claim 1, wherein said stitched images are smoothened, said smoothening comprising gradient blending. 9. The method as claimed in claim 1, wherein said images are further corrected for defects comprising rotation, scaling, shearing, dilations by projective transformation to said images. 10. The method as claimed in claim 1, wherein said desired location for sending the single composite image comprise a database for storing said composite image. 11. The method as claimed in claim 1, wherein said single composite image is transmitted to a user. 12. A system to instantly generate an online image of a document from multiple images captured through a camera equipped mobile device, comprising: a mobile device with image capturing means for capturing multiple images of a document, an application server for receiving said images from the mobile device, means for extracting and obtaining multiple SIFT feature points in a first image and a second image, said first and second images being adjacent, means for matching said feature points obtained in the first image with the feature points obtained in the second image means for aligning said first and second images and stitching said images based on best matching points, and transmitting means for transmitting single composite image to a desired location. 13. The system as claimed in claim 12, wherein said mobile device has transmitting and receiving means for transmitting the captured images to the application server. 14. The system as claimed in claim 12, wherein said mobile device is configured to communicate with the server through a communication service. 15. The system as claimed in claim 12, wherein said mobile device is configured to communicate with the server through a network protocol.

Documents:

1138-del-2006-1138-del-2006-Claims-(13-05-2014).pdf

1138-del-2006-1138-del-2006-Correspondence Others-(13-05-2014).pdf

1138-del-2006-1138-del-2006-Form-2-(13-05-2014).pdf

1138-del-2006-1138-del-2006-GPA-(13-05-2014).pdf

1138-DEL-2006-Abstract-(18-07-2012).pdf

1138-del-2006-abstract.pdf

1138-DEL-2006-Claims-(18-07-2012).pdf

1138-del-2006-claims.pdf

1138-DEL-2006-Correspondence Others-(18-07-2012).pdf

1138-del-2006-Correspondence Others-(30-04-2014).pdf

1138-del-2006-correspondence-others-1.pdf

1138-del-2006-correspondence-others.pdf

1138-del-2006-description (complete).pdf

1138-DEL-2006-Drawings-(18-07-2012).pdf

1138-del-2006-drawings.pdf

1138-del-2006-form-1.pdf

1138-del-2006-form-18.pdf

1138-del-2006-form-2.pdf

1138-del-2006-form-3.pdf

1138-del-2006-form-5.pdf

1138-DEL-2006-GPA-(18-07-2012).pdf

1138-del-2006-GPA-(30-04-2014).pdf

1138-del-2006-gpa.pdf