From 37655cbab153c243eed27781dac10da47052c09b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 31 Jul 2013 13:08:21 +0100 Subject: [PATCH] Updated README. --- README | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 116 insertions(+), 1 deletion(-) diff --git a/README b/README index e392f45..5d0cd75 100644 --- a/README +++ b/README @@ -1,4 +1,119 @@ -POSSIBLY OUTDATED +Exactness is a pixel perfect regression testing tool for Elementary based applications. + +Exactness works by playing back a recording of an interaction with the +application, saving images from that playback, and comparing the images to the +set of "known to be correct" images. Returning a non-zero return value when it +detects images mismatches and creates "error.html", a file that includes the +list of tests failed, including the base images, the new images, and a visual +"diff" of the images. Exactness also fails in case the tests themselves exit +with a non-zero return value, e.g segfault. + += Modes of operation = + +Exactness has four modes of operation, three of which are essential for normal +operation, and the fourth is helpful in some cases, and in general, just pretty. + +== List of Modes == + +# record (-r, --record): record a test macro. +# init (-i, --init): run the test macro and take images of the state. +# play (-p, --play): run the test macro, take images of the state, and compare against the base images (took with init). +# simulation (-s, --simulation): run the test macro, showing the process on screen. + += The List file = + +The list file contains a list of tests to actually run. +The file format is: + + has to be one word (no spaces), there are no restrictions on the command. The command is executed using "system", so you can set environment variables and generally do whatever you'd like. +Empty lines and lines starting with # are ignored. + +Example line: + box ./elm_edje_player tests/box.edj + += Using exactness = + +For the rest of this section we'll assume the name of the list file is +"tests.txt". + +== Recording == + +The first thing to do when running exactness on a new test is generating the +recording file, this is done but running with "-r". New recordings overwrite +old existing recordings so it's always a good idea to backup recordings you are +happy with. +In order to record for a test list run: + exactness -r tests.txt +Just interact with the application in any way you'd like (it's better to have +short interactions as tests run in real time, so running the tests will be as +slow as your recording). +Press F2 to mark an image taking opportunity (as many times you'd like), this +will cause "init" and "play" mode to take images for comparison. Please refer +to the "Limitations" section below to read more about image taking +opportunities. +Running this will create the "base dir" (see command line options for more info +about how to change it) and populate it with one recording file per test. + +== Init == + +After having a recording file for a test you can go on with capturing the base +images. Those images are assumed to be "correct". You should update them when +this assumption is no longer valid (e.g fixed a bug, updated a test). +Run: + exactness -i tests.txt +This will look for recording files in the "base dir", run them, take images of +the application, and put them in "dest-dir/orig". + +== Play == + +This is the most common mode of execution. This should be run on a regular +basis, for example in a CI system. +Run: + exactness -p tests.txt +This will look for recording files in the "base dir", run them, take images of the application, put them in "dest-dir/current", and compare them against the +images in "dest-dir/orig". This will fail (and return a non-zero return value) +if any of the tests return a non-zero return value, or if any of the images +in the "current" dir don't match the ones in "orig" dir. +Detecting a failure is the most important role of exactness, so the return +value is very important. Additionally, an error.html file will be created in +the "current" dir, listing the issues in a human-readable form. + +== Simulation == + +This mode is a show off and a debug mode. Very useful with the "-v" (verbose) +option that shows the mouse cursor on the canvas and outputs information to +console. +Run: + exactness -s tests.txt +or for verbose mode: + exactness -v -s tests.txt +This does nothing other than run the tests to screen. No images are taken, +and no report is generated. + +== Command line options == + +* -b, --base-dir: The location of the recording files. +* -d, --dest-dir: The location of the base dir for the images (orig/current will be appended). +* -w, --wrap: Use a custom command to launch the tests with, e.g valgrind. +* -j, --jobs: Number of jobs to run in parallel. Very useful when running in init and play mode. +* -r, --record: record mode. +* -p, --play: play mode. +* -i, --init: init mode. +* -s, --simulation: simulation mode. +* -v, --verbose: Show cursor location and output extra information to console. +* -h, --help: Help, and information about more boring options. + += Limitations = + +* Animations are very time-specific, you can't really take snapshots of +applications that have animations. +* Exactness is very susceptible to theme/scale/etc. changes, so make sure to +replicate the full environment. +* Only works with Elementary applications. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +REST IS POSSIBLY OUTDATED exactness is a software package aimed to automate Elementary testing after updating elm code.